-- Logs begin at Fri 2022-09-16 02:54:16 UTC, end at Fri 2022-09-16 04:11:20 UTC. -- Sep 16 03:25:53.203245 dv-cinder-857993-1-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 16 03:25:53.215054 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 16 03:25:53.215054 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: *** Starting uWSGI 2.0.18-debian (64bit) on [Fri Sep 16 03:25:53 2022] *** Sep 16 03:25:53.215054 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 16 03:25:53.215054 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 16 03:25:53.215054 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: nodename: dv-cinder-857993-1-iscsi Sep 16 03:25:53.215054 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: machine: x86_64 Sep 16 03:25:53.215054 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: clock source: unix Sep 16 03:25:53.215054 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: pcre jit disabled Sep 16 03:25:53.215054 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: detected number of CPU cores: 4 Sep 16 03:25:53.215054 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: current working directory: / Sep 16 03:25:53.215054 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: detected binary path: /usr/bin/uwsgi-core Sep 16 03:25:53.215054 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: your processes number limit is 47680 Sep 16 03:25:53.215054 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: your memory page size is 4096 bytes Sep 16 03:25:53.215054 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: detected max file descriptor number: 2048 Sep 16 03:25:53.215054 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: lock engine: pthread robust mutexes Sep 16 03:25:53.216398 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: thunder lock: enabled Sep 16 03:25:53.216398 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 16 03:25:53.216398 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 16 03:25:53.254221 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: Python main interpreter initialized at 0x55806a79e9c0 Sep 16 03:25:53.254221 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: python threads support enabled Sep 16 03:25:53.254221 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: your server socket listen backlog is limited to 100 connections Sep 16 03:25:53.254221 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: your mercy for graceful operations on workers is 90 seconds Sep 16 03:25:53.254563 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: mapped 403077 bytes (393 KB) for 2 cores Sep 16 03:25:53.254638 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: *** Operational MODE: preforking *** Sep 16 03:25:53.254933 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: *** uWSGI is running in multiple interpreter mode *** Sep 16 03:25:53.254933 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: spawned uWSGI master process (pid: 81754) Sep 16 03:25:53.255155 dv-cinder-857993-1-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 16 03:25:53.258255 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: spawned uWSGI worker 1 (pid: 81761, cores: 1) Sep 16 03:25:53.258255 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: spawned uWSGI worker 2 (pid: 81762, cores: 1) Sep 16 03:25:53.258255 dv-cinder-857993-1-iscsi devstack@keystone.service[81754]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 16 03:25:57.587395 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 16 03:25:57.587395 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: from cryptography.utils import int_from_bytes Sep 16 03:25:57.588143 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 16 03:25:57.588143 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: from cryptography.utils import int_from_bytes Sep 16 03:25:57.630142 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 16 03:25:57.630142 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: from cryptography.utils import int_from_bytes Sep 16 03:25:57.630683 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 16 03:25:57.630683 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: from cryptography.utils import int_from_bytes Sep 16 03:25:57.637110 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 16 03:25:57.637308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 16 03:25:57.637501 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] command line args: [] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 16 03:25:57.637774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 16 03:25:57.637953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ================================================================================ {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 16 03:25:57.638208 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] admin_token = **** {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.638456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] config_dir = [] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.638643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.638823 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] config_source = [] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.639028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] debug = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.639228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.640421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.640421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.640421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.640421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.640668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] list_limit = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.640668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] log_config_append = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.640784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.640959 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] log_dir = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.641131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] log_file = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.641380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.641616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.641861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.642110 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.642289 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.642466 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.642725 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.643046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.643281 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.643517 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.643752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.643985 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.644232 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.644482 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.644758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.645246 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.49/identity {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.645429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] publish_errors = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.645662 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.645894 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.646073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.646318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.646551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.646786 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] standard_threads = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.647047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.647292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.647451 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.647632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] use_journal = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.647806 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] use_json = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.647982 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] use_stderr = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.648189 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] use_syslog = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.648378 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.648631 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.648863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.649123 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.649361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.649610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.649847 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.650094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.651374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.651374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.651374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.651374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.651374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.651754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.651754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.651982 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.652226 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.652461 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.652715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.653214 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.653466 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.653700 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.653952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.654210 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.654464 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.654712 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.654961 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.655195 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.655424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.655660 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.655909 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.656154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.658127 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.658127 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.658127 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.658127 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.658127 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.658127 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.658127 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.658127 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.659222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.659222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.659362 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.659494 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.659727 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.659968 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.660221 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.660456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.660696 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.660931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.661170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.661409 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.661659 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.661891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.662131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.662369 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.662607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.662844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.663107 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.663384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.663852 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.664220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.664466 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.664703 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.664972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.665216 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.665452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.665691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.665930 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.666202 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.666442 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.666693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.666961 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.667206 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.667447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.667686 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.667936 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.668176 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.668416 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.668661 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.668900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.669148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.669387 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.669624 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.669862 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.670097 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.670334 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.670593 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.670833 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.671102 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.671345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.671583 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.671825 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.672064 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.672300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.672535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.672766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.672997 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.673227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.673462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.673693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.673936 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.674758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.675041 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.675282 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.675540 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.675798 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.676042 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.676276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] auth.external = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.676510 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.676807 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.677042 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.677276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] auth.password = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.677509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] auth.token = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.677750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.677988 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.678233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.678468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.678716 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.678995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.679233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.679472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] credential.caching = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.679720 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.679969 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.680218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.680462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.680707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.680943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.681189 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.681272 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 16 03:25:57.681385 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 16 03:25:57.681469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.681588 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] command line args: [] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 16 03:25:57.681663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.681868 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 16 03:25:57.681945 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.682052 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ================================================================================ {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 16 03:25:57.682178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.682313 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] admin_token = **** {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.682426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.682565 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] config_dir = [] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.682688 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.682756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.682948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.683019 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] config_source = [] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.683136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] debug = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.683205 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.683353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.683459 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] federation.caching = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.683595 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.683674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.683779 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.683897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.684032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.684169 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.684282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.684362 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.684516 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] list_limit = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.684603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.684699 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] log_config_append = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.684826 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.684950 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.685046 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] log_dir = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.685125 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.685225 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] log_file = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.685310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.685462 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.685548 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.685712 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.685791 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.685961 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.686041 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.686210 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.686291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] identity.caching = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.686391 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.686479 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.686572 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.686696 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.686820 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.686951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.687095 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.687185 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.687321 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.687423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.687557 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.687656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.687793 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.687880 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.688027 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.688129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.688278 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.688379 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.688526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.688618 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.688804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.688891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.689047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.689290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.689373 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.49/identity {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.689480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] publish_errors = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.689559 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.689745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.689938 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.690406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.690552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.690677 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.690757 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.690975 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.691109 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.691224 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.691316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.691467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.691553 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] standard_threads = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.691703 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.691783 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.691944 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.692031 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.692126 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.692204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.692308 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] use_journal = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.692387 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.692488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] use_json = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.692630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.692710 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] use_stderr = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.692835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] use_syslog = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.692913 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.693072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 03:25:57.693228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.693319 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.693392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.693557 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.693638 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.693803 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.693882 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.694038 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.694119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.694284 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.694362 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.694523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.694601 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.694751 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.694831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.695014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.695099 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.695246 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.695326 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.695484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.695565 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.695720 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.695808 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.695962 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.696042 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.696200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.696280 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.696436 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.696516 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.696685 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.696764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.696937 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.697056 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.697177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.697291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.697435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.697524 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.697692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.697798 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.697941 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.698023 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.698179 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.698259 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.698437 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.698518 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.698672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.698752 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.698948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.user = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.699033 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.699165 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.699257 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.699424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.699504 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.699655 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.699739 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.699890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.699979 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.700128 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.700209 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.700372 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.700451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.700609 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.700697 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.700841 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.700920 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.701072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.701191 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.701305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.701396 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.701540 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.701621 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.701771 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.701851 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.702017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.702106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.702248 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.702327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.702484 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.702565 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.702728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.702814 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.702982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.703066 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.703220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.703301 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.703453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.703546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.703687 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.703767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.703922 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.704002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.704155 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.704310 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.704388 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.704461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.704617 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.704696 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.704848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.704937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.705086 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.705167 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.705368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.705857 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.705936 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.706123 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.706211 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.706349 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.706438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.706586 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.706667 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.706823 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.706918 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.707089 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.707170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.707328 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.707408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.707562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.707643 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.707792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.707875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.708030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] resource.caching = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.708115 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.708272 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.708353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.708511 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.708591 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.708743 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.708822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.708990 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.709078 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.709225 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.709304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.709457 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.709536 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.709693 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.709774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.709925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.710010 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.710162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.710248 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.710397 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] role.caching = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.710477 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.710643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] role.driver = sql {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.710729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.710873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.711000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.711143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.711261 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.711382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.711463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.711619 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.711699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.711863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.711943 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.712101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.712180 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.712342 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.712423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.712586 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.712666 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.712822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.712901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.713053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.713141 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.713298 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.713377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.713538 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.713620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.713778 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.713866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.714015 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.714101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.714254 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.714428 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.714524 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.714672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.714754 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.714943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.715028 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.715206 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.715287 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.715459 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.715541 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.715700 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.715780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.715935 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.716014 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.716169 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.716249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.716403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.716484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.716636 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.716717 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.716869 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.716950 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.717100 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.717178 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.717332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.717413 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.717561 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.717641 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.717801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.717881 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.718038 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.718119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.718284 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.718364 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.718518 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.718601 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.718746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.718826 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.719004 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] token.caching = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.719087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.719243 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.719323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.719492 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.719572 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] auth.external = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.719722 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.719802 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.719976 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.720055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.720224 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.720304 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.720461 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.720543 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] auth.password = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.720691 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] auth.token = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.720770 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.721457 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.721848 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.721981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.722088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.722238 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.722319 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.722499 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.722579 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.722739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.722820 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.723020 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.723106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.723275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.723356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.723512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.723606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.723754 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] credential.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.723834 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.724008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.724089 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] database.connection = **** {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.724265 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.724394 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.724512 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.724590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.724720 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.724801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.724962 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.725052 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.725197 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.725276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.725432 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.725510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.725671 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.725828 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.725904 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.725978 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.726136 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.726223 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.726371 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.726461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.726614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.726695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.726850 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.726981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.727123 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.727217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.727359 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.727439 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.727603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.727684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] federation.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.727834 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.727923 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.728077 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.728157 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.728312 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.728392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.728465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81761) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 16 03:25:57.728546 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.728775 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.729014 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.729269 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.729506 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.729762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.730001 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.730248 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.730486 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.730743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.731016 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.731256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.731493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.731744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.731977 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.732216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.732467 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.732720 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.732956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.733190 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.733425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.733672 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.733922 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.734170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.734425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.734673 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.734969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.735215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.735456 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.735690 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.735928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.736161 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.736398 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.737695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.737959 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.738221 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.738462 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.738720 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.739001 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.739247 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.739499 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.739750 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.739986 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.740230 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.740464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.740706 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.740951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.741197 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.741438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.741677 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.741930 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.742181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.742417 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.742651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.742928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.743175 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.743415 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.743654 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.743894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.744129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.744370 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.744632 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.744866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.745115 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.745364 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.745613 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.745851 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.746083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.746319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.746555 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.746792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.747049 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.747300 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.747546 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.747793 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.748041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.748287 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.748518 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.748763 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.749000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.749237 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.749473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.749709 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.749953 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.750311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.750558 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.750815 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.751079 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.751331 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.751582 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.751807 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.752057 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.752781 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.753040 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.753282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.753525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.753775 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] resource.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.754033 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.754288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.754530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.754784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.755065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.755308 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.755560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.755800 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.756041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.756283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] role.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.756534 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] role.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.756768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.757019 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.757269 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.757518 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.757766 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.758002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:57.758086 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.758248 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.758327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:57.758472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:57.758563 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.758730 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.759058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.759312 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.759560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.759808 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.760055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.760456 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.760702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.760955 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.761206 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.761454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.761707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.761943 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.762199 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.762453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.762693 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.762972 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.763184 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.763419 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.763671 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.763928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.764178 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.764418 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.764657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.764889 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.765128 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] token.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.765366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.765616 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.765854 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.766110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.766358 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.766596 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.766814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:57.766897 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.767097 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.767347 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.767586 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.768380 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.768628 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.768880 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.769167 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.769402 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.769650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.769917 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.770192 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.connection = **** {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.770441 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.770697 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.770971 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.771222 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.771462 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.771717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.771939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.772177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.772426 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.772664 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.772909 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.773148 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.773399 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.773637 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.773882 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.774114 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.774357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.774595 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 03:25:57.774768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 16 03:25:57.795393 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:57.795622 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:57.801940 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:57.802170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:57.802399 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:57.810803 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:57.829605 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:57.829839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:57.850267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:57.873321 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:57.906614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:57.906852 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:57.913164 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:57.913403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.063493 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.063731 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.063952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.064169 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.064321 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.064417 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.064560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.064706 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.065287 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.065522 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.065747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.065969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.088378 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.088616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.088825 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.088959 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.089088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.089210 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.089332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.089464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.089586 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.089705 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.089823 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.089941 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.089941 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.090177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.108519 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.108677 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.121910 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.122056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.122128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.122332 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.122421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.122565 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.122662 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.122782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.122983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.123270 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.123464 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.123569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.123874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.123986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81761) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 03:25:58.141363 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.141509 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.141797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 16 03:25:58.142009 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 16 03:25:58.142378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.142642 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 16 03:25:58.143418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.143688 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.143989 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.144225 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 16 03:25:58.144328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.144666 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.144881 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.145285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.145690 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.146101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.155262 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.155670 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.156011 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.156376 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.156940 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.157362 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.157773 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.158197 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.164501 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.167533 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81761) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.167641 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.167968 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.168322 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.168678 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.169046 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.169373 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.169710 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.170058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.170397 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.170666 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81761) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.170782 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.171139 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.171347 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.171762 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.172179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.172621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.173032 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.173313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.173604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.173886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.174261 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.174543 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.195401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.196962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.197312 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.197733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.199243 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81761) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.199531 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.202409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.202761 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.203134 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.204247 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81761) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.204538 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.204828 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.210568 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.210934 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.211893 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81761) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.213627 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.215105 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.215461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.215812 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.216164 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.216504 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.218014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81761) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.218318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.218616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.218862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.218988 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.219203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.219289 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.219484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.219599 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.219756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.220193 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.235796 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.235925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.236081 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.236231 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.240017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81761) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.240389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.243277 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.243578 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.243678 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.243915 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.247821 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81761) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.248223 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.250939 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81761) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.251398 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.254749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.255079 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.255170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.255373 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.255461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.255667 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.255764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.255950 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.256046 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.256232 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.256382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.256636 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.271898 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81761) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.272196 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.274094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.274094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.274224 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.274224 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.274224 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.274224 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.279759 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.280788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.281266 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.282492 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.282761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.283087 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.284724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.284996 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.286328 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81761) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.286619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.286939 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.287227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.287614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.287701 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.287913 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.288264 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.288489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.288848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.298751 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81761) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.299197 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.299495 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.299785 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.300174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.300257 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.300468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.300553 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.300748 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.300840 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.301038 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.301323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.301602 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.301922 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.316907 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81761) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.317933 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.320211 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81761) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.321057 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.324619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.324758 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.324835 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.326742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.327462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81761) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.327462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.327462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.327659 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.327659 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.327659 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.333244 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.338274 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.338378 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81761) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.338455 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.338520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.338520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.338520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.338699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.338762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.339059 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.339507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.339795 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.346977 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81761) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.347276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.347570 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.348428 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.348722 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.349005 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.354436 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81761) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.354730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.355038 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.355318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.355596 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.355871 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.355962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 03:25:58.356158 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.356250 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.356436 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.356527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.356706 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.356795 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.356976 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.357077 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.357260 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.357352 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.357553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.357644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.357820 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.357911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.358092 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81761) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.358183 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.358368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.358683 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.358927 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.359243 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.359504 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 03:25:58.393355 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55806a79e9c0 pid: 81761 (default app) Sep 16 03:25:58.394635 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55806a79e9c0 pid: 81762 (default app) Sep 16 03:25:58.396790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a3dc5e-6e16-4afa-b140-3767c38da510 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:25:58.397082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a3dc5e-6e16-4afa-b140-3767c38da510 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:25:58.397358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a3dc5e-6e16-4afa-b140-3767c38da510 None None] PATH_INFO: `/v3` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:25:58.401542 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1/1] 10.222.0.49 () {50 vars in 791 bytes} [Fri Sep 16 03:25:58 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:06.866318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7905914-2a2c-4185-aace-10360664c802 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:06.866934 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7905914-2a2c-4185-aace-10360664c802 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:06.866934 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7905914-2a2c-4185-aace-10360664c802 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:06.871233 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1/2] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:06 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:26:06.876501 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180c7a74-b68e-41c1-9b85-63c9ca51abd1 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:06.877052 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180c7a74-b68e-41c1-9b85-63c9ca51abd1 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:06.877162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180c7a74-b68e-41c1-9b85-63c9ca51abd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:06.950801 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: INFO dbcounter [None req-180c7a74-b68e-41c1-9b85-63c9ca51abd1 None None] Registered counter for database keystone Sep 16 03:26:06.972833 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG oslo_db.sqlalchemy.engines [None req-180c7a74-b68e-41c1-9b85-63c9ca51abd1 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81761) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 16 03:26:06.979226 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writer thread running {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 16 03:26:07.178801 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG passlib.handlers.bcrypt [None req-180c7a74-b68e-41c1-9b85-63c9ca51abd1 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81761) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 16 03:26:07.179154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG passlib.handlers.bcrypt [None req-180c7a74-b68e-41c1-9b85-63c9ca51abd1 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81761) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 16 03:26:07.239591 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-180c7a74-b68e-41c1-9b85-63c9ca51abd1 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:07.241216 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-180c7a74-b68e-41c1-9b85-63c9ca51abd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:07.371433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 2/3] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:06 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 496 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 16 03:26:07.860973 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7905914-2a2c-4185-aace-10360664c802 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:07.930345 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: INFO dbcounter [None req-a7905914-2a2c-4185-aace-10360664c802 None None] Registered counter for database keystone Sep 16 03:26:07.949374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG oslo_db.sqlalchemy.engines [None req-a7905914-2a2c-4185-aace-10360664c802 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81762) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 16 03:26:07.955463 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writer thread running {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 16 03:26:08.073451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7905914-2a2c-4185-aace-10360664c802 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:08.077664 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca7e4b7-8c84-471e-8f41-73ce0e1c49f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:08.078288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca7e4b7-8c84-471e-8f41-73ce0e1c49f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:08.078567 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca7e4b7-8c84-471e-8f41-73ce0e1c49f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:08.078840 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca7e4b7-8c84-471e-8f41-73ce0e1c49f9 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:08.086993 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca7e4b7-8c84-471e-8f41-73ce0e1c49f9 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:08.138984 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: WARNING keystone.common.rbac_enforcer.enforcer [None req-cca7e4b7-8c84-471e-8f41-73ce0e1c49f9 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 16 03:26:08.139284 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca7e4b7-8c84-471e-8f41-73ce0e1c49f9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:08.150545 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 2/4] 10.222.0.49 () {58 vars in 1324 bytes} [Fri Sep 16 03:26:07 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 03:26:08.156727 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180c7a74-b68e-41c1-9b85-63c9ca51abd1 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:08.174719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180c7a74-b68e-41c1-9b85-63c9ca51abd1 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:08.179199 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b1dc4b-b83d-4e95-b4d2-6e2bce05cc24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:08.180313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b1dc4b-b83d-4e95-b4d2-6e2bce05cc24 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:08.180597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b1dc4b-b83d-4e95-b4d2-6e2bce05cc24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:08.180878 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b1dc4b-b83d-4e95-b4d2-6e2bce05cc24 admin admin] PATH_INFO: `/v3/projects/ccdd7a44ba5049fb939dad44d5bd406e` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:08.182441 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b1dc4b-b83d-4e95-b4d2-6e2bce05cc24 admin admin] RBAC: Authorizing `identity:get_project(project_id=ccdd7a44ba5049fb939dad44d5bd406e)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:08.237502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: WARNING keystone.common.rbac_enforcer.enforcer [None req-58b1dc4b-b83d-4e95-b4d2-6e2bce05cc24 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 16 03:26:08.237858 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b1dc4b-b83d-4e95-b4d2-6e2bce05cc24 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:08.242396 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 3/5] 10.222.0.49 () {56 vars in 1207 bytes} [Fri Sep 16 03:26:08 2022] GET /identity/v3/projects/ccdd7a44ba5049fb939dad44d5bd406e => generated 344 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:10.425572 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f39923-c2cd-412e-8c1b-466ab7f7478c None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:10.426182 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f39923-c2cd-412e-8c1b-466ab7f7478c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:10.426182 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f39923-c2cd-412e-8c1b-466ab7f7478c None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:10.430272 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 3/6] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:10 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:26:10.435816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f0fca2-af54-4669-a810-5688eb81ef0b None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:10.436245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f0fca2-af54-4669-a810-5688eb81ef0b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:10.436380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f0fca2-af54-4669-a810-5688eb81ef0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:10.457445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-58f0fca2-af54-4669-a810-5688eb81ef0b None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:10.459000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-58f0fca2-af54-4669-a810-5688eb81ef0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:10.466830 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 4/7] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:10 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 16 03:26:10.731756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f39923-c2cd-412e-8c1b-466ab7f7478c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:10.743887 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f39923-c2cd-412e-8c1b-466ab7f7478c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:10.748149 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83b669b-36cb-4f93-a147-eeef361d8957 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:10.748752 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83b669b-36cb-4f93-a147-eeef361d8957 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:10.749036 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83b669b-36cb-4f93-a147-eeef361d8957 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:10.749315 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83b669b-36cb-4f93-a147-eeef361d8957 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:10.756827 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83b669b-36cb-4f93-a147-eeef361d8957 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:10.758598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83b669b-36cb-4f93-a147-eeef361d8957 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:10.767159 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 4/8] 10.222.0.49 () {58 vars in 1324 bytes} [Fri Sep 16 03:26:10 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 03:26:10.772906 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f0fca2-af54-4669-a810-5688eb81ef0b None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:10.782207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f0fca2-af54-4669-a810-5688eb81ef0b None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:10.787255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db0e6b3-0ec7-4f00-807d-f772ebb5a122 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:10.787883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db0e6b3-0ec7-4f00-807d-f772ebb5a122 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:10.788164 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db0e6b3-0ec7-4f00-807d-f772ebb5a122 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:10.788446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db0e6b3-0ec7-4f00-807d-f772ebb5a122 admin admin] PATH_INFO: `/v3/users/3e71cde4d415471495fcc3441fcdfc54` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:10.789738 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db0e6b3-0ec7-4f00-807d-f772ebb5a122 admin admin] RBAC: Authorizing `identity:get_user(user_id=3e71cde4d415471495fcc3441fcdfc54)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:10.791584 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db0e6b3-0ec7-4f00-807d-f772ebb5a122 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:10.793096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 5/9] 10.222.0.49 () {56 vars in 1198 bytes} [Fri Sep 16 03:26:10 2022] GET /identity/v3/users/3e71cde4d415471495fcc3441fcdfc54 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:13.097907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90a45d7-bc9e-4873-a357-49084e1fde05 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:13.098468 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90a45d7-bc9e-4873-a357-49084e1fde05 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:13.098468 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90a45d7-bc9e-4873-a357-49084e1fde05 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:13.099832 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 5/10] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:26:13.105385 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b638e2-6811-481d-be11-0562f21adbdd None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:13.105869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b638e2-6811-481d-be11-0562f21adbdd None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:13.106490 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b638e2-6811-481d-be11-0562f21adbdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:13.129536 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-06b638e2-6811-481d-be11-0562f21adbdd None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:13.130717 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-06b638e2-6811-481d-be11-0562f21adbdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:13.216856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 6/11] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:13.278726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1919bf5-35b3-407c-9d96-3e3f8d775564 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:13.279027 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1919bf5-35b3-407c-9d96-3e3f8d775564 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:13.279329 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1919bf5-35b3-407c-9d96-3e3f8d775564 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:13.280651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 6/12] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:13.286000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442b2c03-530f-487d-bd01-402c6ebf3f00 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:13.286293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442b2c03-530f-487d-bd01-402c6ebf3f00 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:13.286572 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442b2c03-530f-487d-bd01-402c6ebf3f00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:13.305571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-442b2c03-530f-487d-bd01-402c6ebf3f00 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:13.306758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-442b2c03-530f-487d-bd01-402c6ebf3f00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:13.324305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:13.379080 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 7/13] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:13.487681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1919bf5-35b3-407c-9d96-3e3f8d775564 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:13.628870 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1919bf5-35b3-407c-9d96-3e3f8d775564 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:13.640624 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442b2c03-530f-487d-bd01-402c6ebf3f00 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:13.662946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f15eca-dc73-4431-87f3-35bee14f3a92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:13.663815 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f15eca-dc73-4431-87f3-35bee14f3a92 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:13.664104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f15eca-dc73-4431-87f3-35bee14f3a92 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:13.664384 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f15eca-dc73-4431-87f3-35bee14f3a92 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:13.675561 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f15eca-dc73-4431-87f3-35bee14f3a92 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:13.686022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-07f15eca-dc73-4431-87f3-35bee14f3a92 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:26:13.686022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:13.686022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:13.686022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:13.686022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:13.686022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:13.686022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:13.686022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:13.686022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:13.686022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:13.686022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:13.686022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:13.686022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:13.686022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:13.686022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:13.686022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:13.687065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:13.687065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:13.687065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:13.687065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:13.687065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:13.687065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:13.687065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:13.687065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:13.687065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:13.687065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:13.687065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:13.687065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:13.687065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:13.687065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:13.687065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:13.687065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:13.687907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:13.687907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:13.687907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:13.687907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:13.687907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:13.687907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:13.687907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:13.687907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:13.687907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:13.687907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:13.687907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:26:13.687907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:26:13.687907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 7/14] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:26:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:13.693144 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f15eca-dc73-4431-87f3-35bee14f3a92 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:13.727564 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:13.743498 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442b2c03-530f-487d-bd01-402c6ebf3f00 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:13.777667 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd844448-3a30-4163-af58-e80cd769001f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:13.778555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd844448-3a30-4163-af58-e80cd769001f None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:13.778847 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd844448-3a30-4163-af58-e80cd769001f None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:13.779156 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd844448-3a30-4163-af58-e80cd769001f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:13.796652 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f15eca-dc73-4431-87f3-35bee14f3a92 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:13.800510 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:13.831753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed635c7-7237-458e-945d-5d93d8b7043b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:13.832361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed635c7-7237-458e-945d-5d93d8b7043b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:13.832649 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed635c7-7237-458e-945d-5d93d8b7043b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:13.832932 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed635c7-7237-458e-945d-5d93d8b7043b None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:13.833601 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed635c7-7237-458e-945d-5d93d8b7043b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:13.833910 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed635c7-7237-458e-945d-5d93d8b7043b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:13.835451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed635c7-7237-458e-945d-5d93d8b7043b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:13.845823 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd844448-3a30-4163-af58-e80cd769001f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:13.847314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd844448-3a30-4163-af58-e80cd769001f None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:13.847688 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 8/15] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:26:13.853330 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed635c7-7237-458e-945d-5d93d8b7043b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:13.944654 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 8/16] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:26:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 16 03:26:13.947827 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:13.950581 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd844448-3a30-4163-af58-e80cd769001f None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:13.953813 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed635c7-7237-458e-945d-5d93d8b7043b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:13.989074 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca6c177-bc04-450a-8d31-856edd274d46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:13.989754 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca6c177-bc04-450a-8d31-856edd274d46 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:13.990033 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca6c177-bc04-450a-8d31-856edd274d46 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:13.990309 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca6c177-bc04-450a-8d31-856edd274d46 None admin] PATH_INFO: `/v3/users/3e71cde4d415471495fcc3441fcdfc54` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:13.991859 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca6c177-bc04-450a-8d31-856edd274d46 None admin] RBAC: Authorizing `identity:get_user(user_id=3e71cde4d415471495fcc3441fcdfc54)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:13.993304 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca6c177-bc04-450a-8d31-856edd274d46 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:14.004856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 9/17] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:13 2022] GET /identity/v3/users/3e71cde4d415471495fcc3441fcdfc54 => generated 247 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:26:14.004856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca6c177-bc04-450a-8d31-856edd274d46 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:14.005058 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:14.049437 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd844448-3a30-4163-af58-e80cd769001f None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:14.083338 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708b834e-bd4c-47c7-b2e9-e84c93c65e40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:14.083980 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708b834e-bd4c-47c7-b2e9-e84c93c65e40 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:14.084266 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708b834e-bd4c-47c7-b2e9-e84c93c65e40 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:14.084542 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708b834e-bd4c-47c7-b2e9-e84c93c65e40 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:14.095149 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708b834e-bd4c-47c7-b2e9-e84c93c65e40 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:14.097465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708b834e-bd4c-47c7-b2e9-e84c93c65e40 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:14.101940 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca6c177-bc04-450a-8d31-856edd274d46 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:14.115598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-708b834e-bd4c-47c7-b2e9-e84c93c65e40 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:26:14.115598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:14.115598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:14.115598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:14.115598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:14.115598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:14.115598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:14.115598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:14.115598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:14.115598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:14.115598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:14.115598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:14.115598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:26:14.115598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:26:14.115598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:26:14.115598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:26:14.116769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:14.116769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:14.116769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:14.116769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:14.116769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:14.116769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:14.116769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:14.116769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:14.116769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:14.116769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:14.116769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:14.116769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:14.116769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:14.116769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:14.116769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:14.116769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:14.116769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:26:14.117574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:26:14.117574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:26:14.117574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:14.117574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 9/18] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:26:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:26:14.122645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708b834e-bd4c-47c7-b2e9-e84c93c65e40 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:14.135996 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8b32eb-48c1-412b-a0fd-376fb06e6259 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:14.136632 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8b32eb-48c1-412b-a0fd-376fb06e6259 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:14.136908 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8b32eb-48c1-412b-a0fd-376fb06e6259 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:14.137179 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8b32eb-48c1-412b-a0fd-376fb06e6259 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:14.138376 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8b32eb-48c1-412b-a0fd-376fb06e6259 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:14.139839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8b32eb-48c1-412b-a0fd-376fb06e6259 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:14.141523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 10/19] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:26:13 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:26:14.147045 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8b32eb-48c1-412b-a0fd-376fb06e6259 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:14.178759 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:14.225363 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708b834e-bd4c-47c7-b2e9-e84c93c65e40 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:14.228081 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:14.246326 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8b32eb-48c1-412b-a0fd-376fb06e6259 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:14.259660 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edbc6f4-b315-4c72-aae8-f4533977aa1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:14.260269 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edbc6f4-b315-4c72-aae8-f4533977aa1f None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:14.260553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edbc6f4-b315-4c72-aae8-f4533977aa1f None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:14.260831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edbc6f4-b315-4c72-aae8-f4533977aa1f None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:14.261471 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3edbc6f4-b315-4c72-aae8-f4533977aa1f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:14.261782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3edbc6f4-b315-4c72-aae8-f4533977aa1f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:14.263210 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3edbc6f4-b315-4c72-aae8-f4533977aa1f None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:14.281101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 10/20] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:26:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:26:14.282001 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec32119-3191-4c79-b9d7-d3e8eab1c835 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:14.282620 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec32119-3191-4c79-b9d7-d3e8eab1c835 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:14.282919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec32119-3191-4c79-b9d7-d3e8eab1c835 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:14.283257 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec32119-3191-4c79-b9d7-d3e8eab1c835 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:14.283982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec32119-3191-4c79-b9d7-d3e8eab1c835 None admin] RBAC: Adding query filter params (role.id=89c4b3a971224d468d33d2dc411230a2, scope.domain.id=default, user.id=3e71cde4d415471495fcc3441fcdfc54) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:14.284292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec32119-3191-4c79-b9d7-d3e8eab1c835 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:14.285762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec32119-3191-4c79-b9d7-d3e8eab1c835 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:14.294797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 11/21] 10.222.0.49 () {56 vars in 1322 bytes} [Fri Sep 16 03:26:14 2022] GET /identity/v3/role_assignments?user.id=3e71cde4d415471495fcc3441fcdfc54&scope.domain.id=default&role.id=89c4b3a971224d468d33d2dc411230a2 => generated 233 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:26:16.971767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab084255-eaaa-461c-a2c3-2e7f82049303 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:16.972368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab084255-eaaa-461c-a2c3-2e7f82049303 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:16.972368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab084255-eaaa-461c-a2c3-2e7f82049303 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:16.973966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 11/22] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:16.978859 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71e8480-250d-4bfb-a93a-2c234ef3b3bc None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:16.979342 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71e8480-250d-4bfb-a93a-2c234ef3b3bc None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:16.979427 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71e8480-250d-4bfb-a93a-2c234ef3b3bc None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:16.980897 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 12/23] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:16.986301 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de66196f-8303-41aa-800a-42458f136977 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:16.986625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de66196f-8303-41aa-800a-42458f136977 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:16.986897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de66196f-8303-41aa-800a-42458f136977 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:16.990197 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2be3fc4-0789-4add-ad46-52e84bd4c9dc None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:16.990705 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2be3fc4-0789-4add-ad46-52e84bd4c9dc None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:16.991205 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2be3fc4-0789-4add-ad46-52e84bd4c9dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:17.013365 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG passlib.handlers.bcrypt [None req-d2be3fc4-0789-4add-ad46-52e84bd4c9dc None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81762) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 16 03:26:17.014070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG passlib.handlers.bcrypt [None req-d2be3fc4-0789-4add-ad46-52e84bd4c9dc None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81762) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 16 03:26:17.019893 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-de66196f-8303-41aa-800a-42458f136977 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:17.021118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-de66196f-8303-41aa-800a-42458f136977 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:17.071051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-d2be3fc4-0789-4add-ad46-52e84bd4c9dc None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:17.075128 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d2be3fc4-0789-4add-ad46-52e84bd4c9dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:17.185552 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 12/24] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 03:26:17.225414 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 13/25] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:17.373028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb62d30-29e0-4e6f-a8cb-b0adb8f247ce None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:17.373414 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb62d30-29e0-4e6f-a8cb-b0adb8f247ce None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:17.373590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb62d30-29e0-4e6f-a8cb-b0adb8f247ce None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:17.375209 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 13/26] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:17.381646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1cf028-6d5a-44f3-978d-2b4cfe028951 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:17.382185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1cf028-6d5a-44f3-978d-2b4cfe028951 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:17.382677 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1cf028-6d5a-44f3-978d-2b4cfe028951 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:17.404703 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d772b80-d73b-4258-b44c-b1a21b04eefe None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:17.405327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d772b80-d73b-4258-b44c-b1a21b04eefe None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:17.405822 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d772b80-d73b-4258-b44c-b1a21b04eefe None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:17.407408 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 14/27] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:26:17.410532 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-4a1cf028-6d5a-44f3-978d-2b4cfe028951 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:17.412785 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f9b55a-3478-4195-b6e1-b2d055c5bb8b None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:17.413283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f9b55a-3478-4195-b6e1-b2d055c5bb8b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:17.413779 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f9b55a-3478-4195-b6e1-b2d055c5bb8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:17.419002 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4a1cf028-6d5a-44f3-978d-2b4cfe028951 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:17.442067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-c5f9b55a-3478-4195-b6e1-b2d055c5bb8b None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:17.443632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c5f9b55a-3478-4195-b6e1-b2d055c5bb8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:17.475312 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:17.547350 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 15/28] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 03:26:17.554426 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 14/29] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:17.674672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f9b55a-3478-4195-b6e1-b2d055c5bb8b None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:17.718648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:17.788071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1cf028-6d5a-44f3-978d-2b4cfe028951 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:17.818815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f9b55a-3478-4195-b6e1-b2d055c5bb8b None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:17.873505 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c3e6b9-5236-475a-a134-03763df7e8af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:17.874160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c3e6b9-5236-475a-a134-03763df7e8af None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:17.874446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c3e6b9-5236-475a-a134-03763df7e8af None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:17.874727 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c3e6b9-5236-475a-a134-03763df7e8af None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:17.890669 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c3e6b9-5236-475a-a134-03763df7e8af None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:17.892366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c3e6b9-5236-475a-a134-03763df7e8af None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:17.910174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-26c3e6b9-5236-475a-a134-03763df7e8af None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:26:17.910174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:17.910174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:17.910174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:17.910174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:17.910174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:17.910174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:17.910174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:17.910174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:17.910174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:17.910174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:17.910174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:17.910174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:26:17.910174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:26:17.910174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:26:17.910174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:26:17.911053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:17.911053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:17.911053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:17.911053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:17.911053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:17.911053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:17.911053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:17.911053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:17.911053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:17.911053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:17.911053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:17.911053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:17.911053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:17.911053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:17.911053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:17.911053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:17.911053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:26:17.911871 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:26:17.911871 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:26:17.911871 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:17.911871 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 16/30] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:26:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:17.912244 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c3e6b9-5236-475a-a134-03763df7e8af None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:18.016628 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1cf028-6d5a-44f3-978d-2b4cfe028951 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:18.038245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:18.063881 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c3e6b9-5236-475a-a134-03763df7e8af None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:18.064473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81da57be-221d-4f93-9e9d-e5f6c03aa886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:18.065464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81da57be-221d-4f93-9e9d-e5f6c03aa886 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:18.066015 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81da57be-221d-4f93-9e9d-e5f6c03aa886 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:18.066297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81da57be-221d-4f93-9e9d-e5f6c03aa886 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:18.068166 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81da57be-221d-4f93-9e9d-e5f6c03aa886 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:18.069735 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81da57be-221d-4f93-9e9d-e5f6c03aa886 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:18.106601 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb78b8e7-a230-4a7d-847d-750c0c3e96af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:18.107305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb78b8e7-a230-4a7d-847d-750c0c3e96af None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:18.107552 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb78b8e7-a230-4a7d-847d-750c0c3e96af None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:18.107816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb78b8e7-a230-4a7d-847d-750c0c3e96af None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:18.121504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb78b8e7-a230-4a7d-847d-750c0c3e96af None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:18.128140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-cb78b8e7-a230-4a7d-847d-750c0c3e96af None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:26:18.128140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:18.128140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:18.128140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:18.128140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:18.128140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:18.128140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:18.128140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:18.128140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:18.128140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:18.128140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:18.128140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:18.128140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:18.128140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:18.128140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:18.128140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:18.128993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:18.128993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:18.128993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:18.128993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:18.128993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:18.128993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:18.128993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:18.128993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:18.128993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:18.128993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:18.128993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:18.128993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:18.128993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:18.128993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:18.128993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:18.128993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:18.129864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:18.129864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:18.129864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:18.129864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:18.129864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:18.129864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:18.129864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:18.129864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:18.129864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:18.129864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:18.129864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:26:18.129864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:18.129864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 17/31] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:26:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:18.130517 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb78b8e7-a230-4a7d-847d-750c0c3e96af None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:18.243603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb78b8e7-a230-4a7d-847d-750c0c3e96af None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:18.277785 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc3d95b-1dbd-48ef-b2d3-0950dfd0b036 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:18.278701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc3d95b-1dbd-48ef-b2d3-0950dfd0b036 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:18.279032 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc3d95b-1dbd-48ef-b2d3-0950dfd0b036 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:18.279296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc3d95b-1dbd-48ef-b2d3-0950dfd0b036 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:18.279931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc3d95b-1dbd-48ef-b2d3-0950dfd0b036 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:18.280228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc3d95b-1dbd-48ef-b2d3-0950dfd0b036 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:18.281657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc3d95b-1dbd-48ef-b2d3-0950dfd0b036 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:18.285236 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:18.286691 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 15/32] 10.222.0.49 () {60 vars in 1131 bytes} [Fri Sep 16 03:26:17 2022] POST /identity/v3/roles => generated 219 bytes in 499 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:26:18.287520 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81da57be-221d-4f93-9e9d-e5f6c03aa886 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:18.301943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 18/33] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:26:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:18.303223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc3d95b-1dbd-48ef-b2d3-0950dfd0b036 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:18.328236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:18.419662 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81da57be-221d-4f93-9e9d-e5f6c03aa886 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:18.438530 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc3d95b-1dbd-48ef-b2d3-0950dfd0b036 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:18.463325 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1677bce0-a90a-4383-8952-437616cc8bae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:18.463962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1677bce0-a90a-4383-8952-437616cc8bae None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:18.464244 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1677bce0-a90a-4383-8952-437616cc8bae None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:18.464523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1677bce0-a90a-4383-8952-437616cc8bae None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:18.465210 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1677bce0-a90a-4383-8952-437616cc8bae None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:18.466959 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1677bce0-a90a-4383-8952-437616cc8bae None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:18.485490 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b18b5a-9056-42a2-b870-8220b96d6e87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:18.486118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b18b5a-9056-42a2-b870-8220b96d6e87 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:18.486999 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b18b5a-9056-42a2-b870-8220b96d6e87 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:18.487298 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b18b5a-9056-42a2-b870-8220b96d6e87 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:18.487978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b18b5a-9056-42a2-b870-8220b96d6e87 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:18.489063 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b18b5a-9056-42a2-b870-8220b96d6e87 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:18.490942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b18b5a-9056-42a2-b870-8220b96d6e87 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:18.503880 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 19/34] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:18.505104 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534b0d63-1cbe-4391-9f9d-f0ed5f05d692 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:18.505393 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534b0d63-1cbe-4391-9f9d-f0ed5f05d692 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:18.505703 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534b0d63-1cbe-4391-9f9d-f0ed5f05d692 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:18.507194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 20/35] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:18.507894 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534b0d63-1cbe-4391-9f9d-f0ed5f05d692 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:18.558015 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 16/36] 10.222.0.49 () {60 vars in 1131 bytes} [Fri Sep 16 03:26:18 2022] POST /identity/v3/roles => generated 225 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:26:18.558881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1677bce0-a90a-4383-8952-437616cc8bae None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:18.584219 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:18.648614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534b0d63-1cbe-4391-9f9d-f0ed5f05d692 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:18.678951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:18.692644 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9bd09d-17a8-4c5b-a400-384c525749a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:18.693817 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9bd09d-17a8-4c5b-a400-384c525749a7 None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:18.693817 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9bd09d-17a8-4c5b-a400-384c525749a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:18.693817 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9bd09d-17a8-4c5b-a400-384c525749a7 None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:18.694477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9bd09d-17a8-4c5b-a400-384c525749a7 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:18.695999 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9bd09d-17a8-4c5b-a400-384c525749a7 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:18.708987 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1677bce0-a90a-4383-8952-437616cc8bae None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:18.752188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacd5842-5c92-43ff-a49a-a64f0cec2af6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:18.753227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacd5842-5c92-43ff-a49a-a64f0cec2af6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:18.753672 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacd5842-5c92-43ff-a49a-a64f0cec2af6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:18.754096 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacd5842-5c92-43ff-a49a-a64f0cec2af6 None admin] PATH_INFO: `/v3/users/3e71cde4d415471495fcc3441fcdfc54` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:18.756541 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacd5842-5c92-43ff-a49a-a64f0cec2af6 None admin] RBAC: Authorizing `identity:get_user(user_id=3e71cde4d415471495fcc3441fcdfc54)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:18.759111 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacd5842-5c92-43ff-a49a-a64f0cec2af6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:18.761591 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 17/37] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:18 2022] GET /identity/v3/users/3e71cde4d415471495fcc3441fcdfc54 => generated 247 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:18.763512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47a47c8-0d6d-4147-bc8f-9573dd05d06c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:18.763952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47a47c8-0d6d-4147-bc8f-9573dd05d06c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:18.764371 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47a47c8-0d6d-4147-bc8f-9573dd05d06c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:18.788964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a47a47c8-0d6d-4147-bc8f-9573dd05d06c None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:18.790543 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a47a47c8-0d6d-4147-bc8f-9573dd05d06c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:18.866930 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 18/38] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:18.868100 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fa6732-cdfc-43f8-90b5-740894ea1863 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:18.868387 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fa6732-cdfc-43f8-90b5-740894ea1863 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:18.868663 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fa6732-cdfc-43f8-90b5-740894ea1863 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:18.870081 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 19/39] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:18.870573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28fa6732-cdfc-43f8-90b5-740894ea1863 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:18.942938 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 21/40] 10.222.0.49 () {60 vars in 1141 bytes} [Fri Sep 16 03:26:18 2022] POST /identity/v3/projects => generated 301 bytes in 425 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:26:18.942938 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d796fc-2704-4138-81af-be4ea7b9220e None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:18.942938 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d796fc-2704-4138-81af-be4ea7b9220e None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:18.942938 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d796fc-2704-4138-81af-be4ea7b9220e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:18.963457 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:18.994303 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28fa6732-cdfc-43f8-90b5-740894ea1863 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:19.036880 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3062378c-8e15-47c7-bdde-4818a224fb1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:19.037692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3062378c-8e15-47c7-bdde-4818a224fb1a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:19.038073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3062378c-8e15-47c7-bdde-4818a224fb1a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:19.038469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3062378c-8e15-47c7-bdde-4818a224fb1a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:19.039962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3062378c-8e15-47c7-bdde-4818a224fb1a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:19.041611 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3062378c-8e15-47c7-bdde-4818a224fb1a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:19.043437 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 20/41] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:26:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:19.049944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3062378c-8e15-47c7-bdde-4818a224fb1a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:19.066344 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-82d796fc-2704-4138-81af-be4ea7b9220e None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:19.069890 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-82d796fc-2704-4138-81af-be4ea7b9220e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:19.173688 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3062378c-8e15-47c7-bdde-4818a224fb1a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:19.178801 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 22/42] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 245 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:19.179691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d796fc-2704-4138-81af-be4ea7b9220e None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:19.223294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a3484f-e653-42e6-870f-ec3e56efd13e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:19.224009 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a3484f-e653-42e6-870f-ec3e56efd13e None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:19.224299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a3484f-e653-42e6-870f-ec3e56efd13e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:19.224613 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a3484f-e653-42e6-870f-ec3e56efd13e None admin] PATH_INFO: `/v3/domains/default/users/3e71cde4d415471495fcc3441fcdfc54/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:19.228206 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a3484f-e653-42e6-870f-ec3e56efd13e None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=3e71cde4d415471495fcc3441fcdfc54, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:19.229694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a3484f-e653-42e6-870f-ec3e56efd13e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:19.234577 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:19.291566 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 21/43] 10.222.0.49 () {58 vars in 1361 bytes} [Fri Sep 16 03:26:19 2022] PUT /identity/v3/domains/default/users/3e71cde4d415471495fcc3441fcdfc54/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:26:19.304396 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d796fc-2704-4138-81af-be4ea7b9220e None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:19.349558 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20cf448-e2d4-41b5-9665-e4e4b6b98794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:19.350211 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20cf448-e2d4-41b5-9665-e4e4b6b98794 None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:19.350503 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20cf448-e2d4-41b5-9665-e4e4b6b98794 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:19.350782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20cf448-e2d4-41b5-9665-e4e4b6b98794 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:19.351545 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20cf448-e2d4-41b5-9665-e4e4b6b98794 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:19.353251 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20cf448-e2d4-41b5-9665-e4e4b6b98794 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:19.407242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 23/44] 10.222.0.49 () {60 vars in 1131 bytes} [Fri Sep 16 03:26:19 2022] POST /identity/v3/roles => generated 223 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:26:19.455976 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a3484f-e653-42e6-870f-ec3e56efd13e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:19.459104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:19.570668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a3484f-e653-42e6-870f-ec3e56efd13e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:19.607448 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03efad3f-6094-44a9-8fb1-747da18da045 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:19.608095 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03efad3f-6094-44a9-8fb1-747da18da045 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:19.608384 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03efad3f-6094-44a9-8fb1-747da18da045 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:19.608716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03efad3f-6094-44a9-8fb1-747da18da045 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:19.609840 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03efad3f-6094-44a9-8fb1-747da18da045 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:19.611421 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03efad3f-6094-44a9-8fb1-747da18da045 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:19.613121 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 22/45] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:26:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:19.618776 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20cf448-e2d4-41b5-9665-e4e4b6b98794 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:19.717472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:19.720305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20cf448-e2d4-41b5-9665-e4e4b6b98794 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:19.754807 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468cf1f7-078f-4897-8c07-f37c53a1e042 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:19.755450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468cf1f7-078f-4897-8c07-f37c53a1e042 None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:19.755734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468cf1f7-078f-4897-8c07-f37c53a1e042 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:19.756012 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468cf1f7-078f-4897-8c07-f37c53a1e042 None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:19.756672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468cf1f7-078f-4897-8c07-f37c53a1e042 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:19.758137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468cf1f7-078f-4897-8c07-f37c53a1e042 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:19.829509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 24/46] 10.222.0.49 () {60 vars in 1141 bytes} [Fri Sep 16 03:26:19 2022] POST /identity/v3/projects => generated 312 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:26:21.449220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec734ee-c597-45cf-9ecd-945c6f4088dc None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:21.449728 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec734ee-c597-45cf-9ecd-945c6f4088dc None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:21.449815 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec734ee-c597-45cf-9ecd-945c6f4088dc None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:21.451127 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 23/47] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:21.456357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9467814-d79b-4747-b550-7a7daece340f None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:21.456784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9467814-d79b-4747-b550-7a7daece340f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:21.456984 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9467814-d79b-4747-b550-7a7daece340f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:21.475352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-a9467814-d79b-4747-b550-7a7daece340f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:21.476520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a9467814-d79b-4747-b550-7a7daece340f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:21.542588 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 25/48] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:21.811750 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec734ee-c597-45cf-9ecd-945c6f4088dc None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:21.883180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:21.916459 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec734ee-c597-45cf-9ecd-945c6f4088dc None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:21.952427 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033ac677-767d-4f4c-b27e-154e1f360f74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:21.953210 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033ac677-767d-4f4c-b27e-154e1f360f74 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:21.953390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033ac677-767d-4f4c-b27e-154e1f360f74 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:21.953667 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033ac677-767d-4f4c-b27e-154e1f360f74 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:21.964286 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033ac677-767d-4f4c-b27e-154e1f360f74 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:21.969556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-033ac677-767d-4f4c-b27e-154e1f360f74 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:26:21.969556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:21.969556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:21.969556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:21.969556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:21.969556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:21.969556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:21.969556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:21.969556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:21.969556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:21.969556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:21.969556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:21.969556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:21.969556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:21.969556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:21.969556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:21.970404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:21.970404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:21.970404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:21.970404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:21.970404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:21.970404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:21.970404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:21.970404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:21.970404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:21.970404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:21.970404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:21.970404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:21.970404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:21.970404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:21.970404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:21.970404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:21.971716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:21.971716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:21.971716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:21.971716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:21.971716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:21.971716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:21.971716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:21.971716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:21.971716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:21.971716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:21.971716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:26:21.971716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:26:21.971716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 24/49] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:26:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:21.976954 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9467814-d79b-4747-b550-7a7daece340f None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:22.021172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c3cf5d-fd3f-42c0-a662-4b6ebe77b14d None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:22.021456 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c3cf5d-fd3f-42c0-a662-4b6ebe77b14d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:22.021737 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c3cf5d-fd3f-42c0-a662-4b6ebe77b14d None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:22.023112 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 25/50] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:22.028036 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbc20f6-d25f-4d12-b23f-d3f7719831ee None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:22.028323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbc20f6-d25f-4d12-b23f-d3f7719831ee None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:22.028858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbc20f6-d25f-4d12-b23f-d3f7719831ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:22.047443 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-6dbc20f6-d25f-4d12-b23f-d3f7719831ee None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:22.049200 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6dbc20f6-d25f-4d12-b23f-d3f7719831ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:22.059243 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:22.077395 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9467814-d79b-4747-b550-7a7daece340f None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:22.114048 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfefcc25-39b4-4fdf-a1d8-a88a3220cd1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:22.114669 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfefcc25-39b4-4fdf-a1d8-a88a3220cd1e None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:22.114996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfefcc25-39b4-4fdf-a1d8-a88a3220cd1e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:22.115272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfefcc25-39b4-4fdf-a1d8-a88a3220cd1e None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:22.116038 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 26/51] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 03:26:22.116148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfefcc25-39b4-4fdf-a1d8-a88a3220cd1e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:22.116232 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfefcc25-39b4-4fdf-a1d8-a88a3220cd1e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:22.117645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfefcc25-39b4-4fdf-a1d8-a88a3220cd1e None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:22.125750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 26/52] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:22.131972 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbc20f6-d25f-4d12-b23f-d3f7719831ee None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:22.191290 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:22.230412 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbc20f6-d25f-4d12-b23f-d3f7719831ee None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:22.262791 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0be3399-0b1c-4803-be3c-6046ace5c3f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:22.263444 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0be3399-0b1c-4803-be3c-6046ace5c3f1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:22.263727 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0be3399-0b1c-4803-be3c-6046ace5c3f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:22.264004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0be3399-0b1c-4803-be3c-6046ace5c3f1 None admin] PATH_INFO: `/v3/users/3e71cde4d415471495fcc3441fcdfc54` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:22.265154 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0be3399-0b1c-4803-be3c-6046ace5c3f1 None admin] RBAC: Authorizing `identity:get_user(user_id=3e71cde4d415471495fcc3441fcdfc54)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:22.266625 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0be3399-0b1c-4803-be3c-6046ace5c3f1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:22.268264 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 27/53] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:22 2022] GET /identity/v3/users/3e71cde4d415471495fcc3441fcdfc54 => generated 247 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:22.273877 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfefcc25-39b4-4fdf-a1d8-a88a3220cd1e None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:22.375403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfefcc25-39b4-4fdf-a1d8-a88a3220cd1e None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:22.388071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0be3399-0b1c-4803-be3c-6046ace5c3f1 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:22.410225 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc06403-6438-49df-8b3b-b18187a9a559 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:22.410858 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc06403-6438-49df-8b3b-b18187a9a559 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:22.411167 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc06403-6438-49df-8b3b-b18187a9a559 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:22.411443 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc06403-6438-49df-8b3b-b18187a9a559 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:22.412543 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc06403-6438-49df-8b3b-b18187a9a559 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:22.413977 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc06403-6438-49df-8b3b-b18187a9a559 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:22.415542 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 27/54] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:26:22 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:22.421364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc06403-6438-49df-8b3b-b18187a9a559 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:22.435937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:22.486868 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0be3399-0b1c-4803-be3c-6046ace5c3f1 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:22.512750 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:22.518528 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc06403-6438-49df-8b3b-b18187a9a559 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:22.520970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c20fbb9-da6c-465c-8060-192bd2cc6a36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:22.521601 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c20fbb9-da6c-465c-8060-192bd2cc6a36 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:22.521883 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c20fbb9-da6c-465c-8060-192bd2cc6a36 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:22.522156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c20fbb9-da6c-465c-8060-192bd2cc6a36 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:22.523288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c20fbb9-da6c-465c-8060-192bd2cc6a36 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:22.524817 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c20fbb9-da6c-465c-8060-192bd2cc6a36 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:22.526366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 28/55] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:26:22 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:26:22.532279 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c20fbb9-da6c-465c-8060-192bd2cc6a36 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:22.555904 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d91ee4f-e3fe-4ea6-9c5b-16201a14f29e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:22.556518 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d91ee4f-e3fe-4ea6-9c5b-16201a14f29e None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:22.556798 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d91ee4f-e3fe-4ea6-9c5b-16201a14f29e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:22.557072 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d91ee4f-e3fe-4ea6-9c5b-16201a14f29e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:22.557798 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d91ee4f-e3fe-4ea6-9c5b-16201a14f29e None admin] RBAC: Adding query filter params (role.id=89c4b3a971224d468d33d2dc411230a2, scope.domain.id=default, user.id=3e71cde4d415471495fcc3441fcdfc54) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:22.558146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d91ee4f-e3fe-4ea6-9c5b-16201a14f29e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:22.559631 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d91ee4f-e3fe-4ea6-9c5b-16201a14f29e None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:22.566728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 28/56] 10.222.0.49 () {56 vars in 1322 bytes} [Fri Sep 16 03:26:22 2022] GET /identity/v3/role_assignments?user.id=3e71cde4d415471495fcc3441fcdfc54&scope.domain.id=default&role.id=89c4b3a971224d468d33d2dc411230a2 => generated 530 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:26:22.743729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c20fbb9-da6c-465c-8060-192bd2cc6a36 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:22.778828 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007f21c9-166b-42fb-82bf-6ce7321f7088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:22.779500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007f21c9-166b-42fb-82bf-6ce7321f7088 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:22.779782 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007f21c9-166b-42fb-82bf-6ce7321f7088 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:22.780055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007f21c9-166b-42fb-82bf-6ce7321f7088 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:22.780706 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007f21c9-166b-42fb-82bf-6ce7321f7088 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:22.782142 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007f21c9-166b-42fb-82bf-6ce7321f7088 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:22.916060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 29/57] 10.222.0.49 () {60 vars in 1140 bytes} [Fri Sep 16 03:26:22 2022] POST /identity/v3/projects => generated 298 bytes in 384 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:26:25.074633 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248a1466-c1da-45c7-ab81-388f80ce4fe0 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:25.075203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248a1466-c1da-45c7-ab81-388f80ce4fe0 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:25.075203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248a1466-c1da-45c7-ab81-388f80ce4fe0 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:25.076564 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 29/58] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:25.082178 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8f1840-f075-40da-b19f-399f3d83443d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:25.082558 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8f1840-f075-40da-b19f-399f3d83443d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:25.082730 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8f1840-f075-40da-b19f-399f3d83443d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:25.100866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9b8f1840-f075-40da-b19f-399f3d83443d None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:25.102047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9b8f1840-f075-40da-b19f-399f3d83443d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:25.143655 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:25.166886 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 30/59] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:25.434740 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248a1466-c1da-45c7-ab81-388f80ce4fe0 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:25.513545 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:25.537017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248a1466-c1da-45c7-ab81-388f80ce4fe0 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:25.570251 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770c9f1c-6a3c-434f-b9ce-9f68e0b025c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:25.570879 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770c9f1c-6a3c-434f-b9ce-9f68e0b025c4 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:25.571195 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770c9f1c-6a3c-434f-b9ce-9f68e0b025c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:25.571474 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770c9f1c-6a3c-434f-b9ce-9f68e0b025c4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:25.572586 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770c9f1c-6a3c-434f-b9ce-9f68e0b025c4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:25.574033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770c9f1c-6a3c-434f-b9ce-9f68e0b025c4 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:25.575626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 30/60] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:26:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:25.581450 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8f1840-f075-40da-b19f-399f3d83443d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:25.676160 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8f1840-f075-40da-b19f-399f3d83443d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:25.708516 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a26b37e-a759-4a62-ae5c-628002f6c14d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:25.709120 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a26b37e-a759-4a62-ae5c-628002f6c14d None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:25.709403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a26b37e-a759-4a62-ae5c-628002f6c14d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:25.709678 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a26b37e-a759-4a62-ae5c-628002f6c14d None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:25.710317 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a26b37e-a759-4a62-ae5c-628002f6c14d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:25.711785 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a26b37e-a759-4a62-ae5c-628002f6c14d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:25.804693 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 31/61] 10.222.0.49 () {60 vars in 1132 bytes} [Fri Sep 16 03:26:25 2022] POST /identity/v3/users => generated 275 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:26:29.108304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d000b1c3-95b6-4ab2-b111-c5708275fbeb None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:29.108820 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d000b1c3-95b6-4ab2-b111-c5708275fbeb None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:29.108927 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d000b1c3-95b6-4ab2-b111-c5708275fbeb None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:29.110489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 31/62] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:29.124956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f8b6e6-1278-406a-9ba5-eeb762c6779a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:29.125814 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f8b6e6-1278-406a-9ba5-eeb762c6779a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:29.126240 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f8b6e6-1278-406a-9ba5-eeb762c6779a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:29.156030 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-47f8b6e6-1278-406a-9ba5-eeb762c6779a None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:29.157489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-47f8b6e6-1278-406a-9ba5-eeb762c6779a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:29.194869 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:29.247871 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c0de8a-5f91-4484-85b7-edd79df53c11 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:29.248455 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 32/63] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:29.248951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c0de8a-5f91-4484-85b7-edd79df53c11 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:29.249438 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c0de8a-5f91-4484-85b7-edd79df53c11 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:29.251072 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 32/64] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:29.257266 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3097c7-15a3-4db3-8b13-780a50b4bf1e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:29.257563 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3097c7-15a3-4db3-8b13-780a50b4bf1e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:29.257848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3097c7-15a3-4db3-8b13-780a50b4bf1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:29.280187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-6d3097c7-15a3-4db3-8b13-780a50b4bf1e None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:29.281392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6d3097c7-15a3-4db3-8b13-780a50b4bf1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:29.366649 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 33/65] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:29.569326 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c0de8a-5f91-4484-85b7-edd79df53c11 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:29.654658 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3097c7-15a3-4db3-8b13-780a50b4bf1e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:29.709349 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c0de8a-5f91-4484-85b7-edd79df53c11 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:29.757486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca4681d-96cb-445c-acd1-02ecd7e912d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:29.759164 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca4681d-96cb-445c-acd1-02ecd7e912d8 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:29.759613 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca4681d-96cb-445c-acd1-02ecd7e912d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:29.760026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca4681d-96cb-445c-acd1-02ecd7e912d8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:29.774029 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca4681d-96cb-445c-acd1-02ecd7e912d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:29.780685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-bca4681d-96cb-445c-acd1-02ecd7e912d8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:26:29.780685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:29.780685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:29.780685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:29.780685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:29.780685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:29.780685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:29.780685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:29.780685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:29.780685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:29.780685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:29.780685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:29.780685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:29.780685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:29.780685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:29.780685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:29.781647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:29.781647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:29.781647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:29.781647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:29.781647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:29.781647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:29.781647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:29.781647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:29.781647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:29.781647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:29.781647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:29.781647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:29.781647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:29.781647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:29.781647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:29.781647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:29.782472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:29.782472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:29.782472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:29.782472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:29.782472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:29.782472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:29.782472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:29.782472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:29.782472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:29.782472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:29.782472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:26:29.782472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:29.783621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 33/66] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:29 2022] GET /identity/v3/roles/member => generated 84 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:29.784443 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546abf0b-29d6-4bc0-bbd5-ffe03bd1b737 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:29.784732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546abf0b-29d6-4bc0-bbd5-ffe03bd1b737 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:29.785009 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546abf0b-29d6-4bc0-bbd5-ffe03bd1b737 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:29.786395 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 34/67] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:29.787315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5f1e5f-e990-4d7e-aa5c-3cbb39426f5f None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:29.787603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5f1e5f-e990-4d7e-aa5c-3cbb39426f5f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:29.787881 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5f1e5f-e990-4d7e-aa5c-3cbb39426f5f None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:29.789221 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 35/68] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:29 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:29.790130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:29.795793 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5f1e5f-e990-4d7e-aa5c-3cbb39426f5f None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:29.799730 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3097c7-15a3-4db3-8b13-780a50b4bf1e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:29.801421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:29.833624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b4c6af-4f1e-4141-857b-b3e702fadf40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:29.834274 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b4c6af-4f1e-4141-857b-b3e702fadf40 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:29.834558 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b4c6af-4f1e-4141-857b-b3e702fadf40 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:29.834835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b4c6af-4f1e-4141-857b-b3e702fadf40 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:29.845111 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b4c6af-4f1e-4141-857b-b3e702fadf40 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:29.850769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-01b4c6af-4f1e-4141-857b-b3e702fadf40 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:26:29.850769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:29.850769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:29.850769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:29.850769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:29.850769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:29.850769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:29.850769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:29.850769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:29.850769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:29.850769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:29.850769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:29.850769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:29.850769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:29.850769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:29.850769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:29.851824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:29.851824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:29.851824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:29.851824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:29.851824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:29.851824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:29.851824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:29.851824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:29.851824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:29.851824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:29.851824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:29.851824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:29.851824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:29.851824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:29.851824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:29.851824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:29.852874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:29.852874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:29.852874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:29.852874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:29.852874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:29.852874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:29.852874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:29.852874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:29.852874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:29.852874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:29.852874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:26:29.852874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:26:29.852874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 34/69] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:26:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:26:29.855299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921dec5f-6637-476d-a32f-48832fea0757 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:29.855588 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921dec5f-6637-476d-a32f-48832fea0757 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:29.855866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921dec5f-6637-476d-a32f-48832fea0757 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:29.873798 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-921dec5f-6637-476d-a32f-48832fea0757 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:29.875021 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-921dec5f-6637-476d-a32f-48832fea0757 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:29.901096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5f1e5f-e990-4d7e-aa5c-3cbb39426f5f None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:29.934568 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078f28be-082c-4ba1-a636-a08d5ae602c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:29.935139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078f28be-082c-4ba1-a636-a08d5ae602c2 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:29.935425 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078f28be-082c-4ba1-a636-a08d5ae602c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:29.935707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078f28be-082c-4ba1-a636-a08d5ae602c2 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:29.936362 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078f28be-082c-4ba1-a636-a08d5ae602c2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:29.936665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078f28be-082c-4ba1-a636-a08d5ae602c2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:29.938055 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078f28be-082c-4ba1-a636-a08d5ae602c2 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:29.940971 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 35/70] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:29.942196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741bf571-2ae5-44ff-a21c-5210d7abe2f2 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:29.942486 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741bf571-2ae5-44ff-a21c-5210d7abe2f2 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:29.942770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741bf571-2ae5-44ff-a21c-5210d7abe2f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:29.952119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 36/71] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:26:29 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:29.952119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f504968e-1691-431c-9e9b-1f93f7164499 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:29.952119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f504968e-1691-431c-9e9b-1f93f7164499 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:29.952119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f504968e-1691-431c-9e9b-1f93f7164499 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:29.952119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 37/72] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:29 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:29.952119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f504968e-1691-431c-9e9b-1f93f7164499 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:29.967032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-741bf571-2ae5-44ff-a21c-5210d7abe2f2 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:29.968658 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-741bf571-2ae5-44ff-a21c-5210d7abe2f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:30.020783 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:30.035818 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 36/73] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:30.044332 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741bf571-2ae5-44ff-a21c-5210d7abe2f2 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:30.061128 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:30.067038 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f504968e-1691-431c-9e9b-1f93f7164499 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:30.111308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ae6c92-8858-4994-93cf-a36bdb77811b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:30.111926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ae6c92-8858-4994-93cf-a36bdb77811b None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:30.112212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ae6c92-8858-4994-93cf-a36bdb77811b None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:30.112613 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ae6c92-8858-4994-93cf-a36bdb77811b None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:30.113424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ae6c92-8858-4994-93cf-a36bdb77811b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:30.113727 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ae6c92-8858-4994-93cf-a36bdb77811b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:30.115327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ae6c92-8858-4994-93cf-a36bdb77811b None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:30.125683 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 38/74] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:30.130488 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22adbd4-2b63-4adf-8000-b0d452de3aa2 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:30.130488 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22adbd4-2b63-4adf-8000-b0d452de3aa2 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:30.130488 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22adbd4-2b63-4adf-8000-b0d452de3aa2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:30.150669 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-e22adbd4-2b63-4adf-8000-b0d452de3aa2 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:30.151994 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e22adbd4-2b63-4adf-8000-b0d452de3aa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:30.166157 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741bf571-2ae5-44ff-a21c-5210d7abe2f2 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:30.205800 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904a26b6-70af-4fd3-9fdc-39f22df925f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:30.206766 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904a26b6-70af-4fd3-9fdc-39f22df925f3 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:30.207257 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904a26b6-70af-4fd3-9fdc-39f22df925f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:30.207734 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904a26b6-70af-4fd3-9fdc-39f22df925f3 None admin] PATH_INFO: `/v3/users/5eef2e8ac0a34e458d28dc3d163cb041` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:30.223931 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904a26b6-70af-4fd3-9fdc-39f22df925f3 None admin] RBAC: Authorizing `identity:get_user(user_id=5eef2e8ac0a34e458d28dc3d163cb041)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:30.225403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904a26b6-70af-4fd3-9fdc-39f22df925f3 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:30.227228 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 37/75] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:30 2022] GET /identity/v3/users/5eef2e8ac0a34e458d28dc3d163cb041 => generated 275 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:30.227866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904a26b6-70af-4fd3-9fdc-39f22df925f3 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:30.238512 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 39/76] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:30.238512 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22adbd4-2b63-4adf-8000-b0d452de3aa2 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:30.306294 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:30.321028 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:30.351774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22adbd4-2b63-4adf-8000-b0d452de3aa2 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:30.352265 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904a26b6-70af-4fd3-9fdc-39f22df925f3 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:30.387399 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed14cbbc-f954-48f4-9e7f-d921b891f1c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:30.388288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed14cbbc-f954-48f4-9e7f-d921b891f1c4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:30.388572 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed14cbbc-f954-48f4-9e7f-d921b891f1c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:30.388849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed14cbbc-f954-48f4-9e7f-d921b891f1c4 None admin] PATH_INFO: `/v3/users/3e71cde4d415471495fcc3441fcdfc54` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:30.389912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae07865-e2a6-4749-9fc3-04fd4b9d15c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:30.390094 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed14cbbc-f954-48f4-9e7f-d921b891f1c4 None admin] RBAC: Authorizing `identity:get_user(user_id=3e71cde4d415471495fcc3441fcdfc54)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:30.390551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae07865-e2a6-4749-9fc3-04fd4b9d15c3 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:30.390833 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae07865-e2a6-4749-9fc3-04fd4b9d15c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:30.391137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae07865-e2a6-4749-9fc3-04fd4b9d15c3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:30.391550 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed14cbbc-f954-48f4-9e7f-d921b891f1c4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:30.394249 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 38/77] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:30 2022] GET /identity/v3/users/3e71cde4d415471495fcc3441fcdfc54 => generated 247 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:30.399189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed14cbbc-f954-48f4-9e7f-d921b891f1c4 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:30.406029 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae07865-e2a6-4749-9fc3-04fd4b9d15c3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:30.411947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-0ae07865-e2a6-4749-9fc3-04fd4b9d15c3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:26:30.411947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:30.411947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:30.411947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:30.411947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:30.411947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:30.411947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:30.411947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:30.411947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:30.411947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:30.411947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:30.411947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:30.411947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:30.411947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:30.411947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:30.411947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:30.412813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:30.412813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:30.412813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:30.412813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:30.412813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:30.412813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:30.412813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:30.412813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:30.412813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:30.412813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:30.412813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:30.412813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:30.412813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:30.412813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:30.412813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:30.412813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:30.413573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:30.413573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:30.413573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:30.413573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:30.413573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:30.413573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:30.413573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:30.413573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:30.413573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:30.413573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:30.413573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:26:30.413573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:30.413573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 40/78] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:30 2022] GET /identity/v3/roles/member => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:30.414944 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae07865-e2a6-4749-9fc3-04fd4b9d15c3 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:30.508028 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed14cbbc-f954-48f4-9e7f-d921b891f1c4 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:30.519448 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae07865-e2a6-4749-9fc3-04fd4b9d15c3 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:30.537125 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:30.542616 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e82a32-36b6-45ad-8ab5-ba4fd03f327d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:30.543263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e82a32-36b6-45ad-8ab5-ba4fd03f327d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:30.543546 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e82a32-36b6-45ad-8ab5-ba4fd03f327d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:30.543817 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e82a32-36b6-45ad-8ab5-ba4fd03f327d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:30.553576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0594c1d2-af71-4156-9166-b3e6281a9fa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:30.554209 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0594c1d2-af71-4156-9166-b3e6281a9fa0 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:30.554498 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0594c1d2-af71-4156-9166-b3e6281a9fa0 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:30.554613 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:30.554772 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0594c1d2-af71-4156-9166-b3e6281a9fa0 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:30.564746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0594c1d2-af71-4156-9166-b3e6281a9fa0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:30.569595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-0594c1d2-af71-4156-9166-b3e6281a9fa0 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:26:30.569595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:30.569595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:30.569595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:30.569595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:30.569595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:30.569595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:30.569595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:30.569595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:30.569595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:30.569595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:30.569595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:30.569595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:30.569595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:30.569595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:30.569595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:30.570615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:30.570615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:30.570615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:30.570615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:30.570615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:30.570615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:30.570615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:30.570615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:30.570615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:30.570615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:30.570615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:30.570615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:30.570615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:30.570615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:30.570615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:30.570615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:30.571455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:30.571455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:30.571455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:30.571455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:30.571455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:30.571455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:30.571455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:30.571455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:30.571455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:30.571455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:30.571455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:26:30.571455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:30.573587 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 41/79] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:26:30 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:30.573587 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0594c1d2-af71-4156-9166-b3e6281a9fa0 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:30.612643 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e82a32-36b6-45ad-8ab5-ba4fd03f327d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:30.614117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e82a32-36b6-45ad-8ab5-ba4fd03f327d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:30.671420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0594c1d2-af71-4156-9166-b3e6281a9fa0 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:30.706034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7dafa3-12e6-4e6e-95a5-7ff3a0d85bdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:30.706889 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7dafa3-12e6-4e6e-95a5-7ff3a0d85bdb None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:30.707212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7dafa3-12e6-4e6e-95a5-7ff3a0d85bdb None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:30.707491 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7dafa3-12e6-4e6e-95a5-7ff3a0d85bdb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:30.710287 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 39/80] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:26:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:26:30.711159 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e82a32-36b6-45ad-8ab5-ba4fd03f327d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:30.750727 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:30.754095 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:30.772537 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7dafa3-12e6-4e6e-95a5-7ff3a0d85bdb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:30.773958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7dafa3-12e6-4e6e-95a5-7ff3a0d85bdb None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:30.809668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e82a32-36b6-45ad-8ab5-ba4fd03f327d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:30.842894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10680266-ac26-4923-a8c0-e4a9c41ae3e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:30.843536 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10680266-ac26-4923-a8c0-e4a9c41ae3e1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:30.843822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10680266-ac26-4923-a8c0-e4a9c41ae3e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:30.844099 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10680266-ac26-4923-a8c0-e4a9c41ae3e1 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:30.844760 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10680266-ac26-4923-a8c0-e4a9c41ae3e1 None admin] RBAC: Adding query filter params (name=member) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:30.845069 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10680266-ac26-4923-a8c0-e4a9c41ae3e1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:30.846589 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10680266-ac26-4923-a8c0-e4a9c41ae3e1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:30.854683 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 40/81] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:26:30 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:30.855502 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10680266-ac26-4923-a8c0-e4a9c41ae3e1 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:30.871639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 42/82] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:26:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:26:30.872601 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7dafa3-12e6-4e6e-95a5-7ff3a0d85bdb None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:30.948136 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:30.956677 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10680266-ac26-4923-a8c0-e4a9c41ae3e1 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:30.958254 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:30.968923 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7dafa3-12e6-4e6e-95a5-7ff3a0d85bdb None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:30.990378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051b140e-a03d-481c-a5f4-e362ed7fb3b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:30.991042 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051b140e-a03d-481c-a5f4-e362ed7fb3b1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:30.991326 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051b140e-a03d-481c-a5f4-e362ed7fb3b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:30.991600 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051b140e-a03d-481c-a5f4-e362ed7fb3b1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:30.992694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051b140e-a03d-481c-a5f4-e362ed7fb3b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:30.994140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051b140e-a03d-481c-a5f4-e362ed7fb3b1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:30.995921 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 41/83] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:26:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:30.996541 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051b140e-a03d-481c-a5f4-e362ed7fb3b1 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:31.003146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb3376a-38db-4018-82a3-ce1405c8c5fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:31.003774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb3376a-38db-4018-82a3-ce1405c8c5fd None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:31.004060 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb3376a-38db-4018-82a3-ce1405c8c5fd None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:31.004340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb3376a-38db-4018-82a3-ce1405c8c5fd None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:31.005018 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb3376a-38db-4018-82a3-ce1405c8c5fd None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:31.005328 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb3376a-38db-4018-82a3-ce1405c8c5fd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:31.006707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb3376a-38db-4018-82a3-ce1405c8c5fd None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:31.019853 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 43/84] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:26:30 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:31.019853 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb3376a-38db-4018-82a3-ce1405c8c5fd None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:31.099263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051b140e-a03d-481c-a5f4-e362ed7fb3b1 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:31.114198 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb3376a-38db-4018-82a3-ce1405c8c5fd None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:31.133689 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9b4d67-805a-4456-a23b-3f6ab0420e01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:31.134321 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9b4d67-805a-4456-a23b-3f6ab0420e01 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:31.134604 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9b4d67-805a-4456-a23b-3f6ab0420e01 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:31.134885 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9b4d67-805a-4456-a23b-3f6ab0420e01 None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:31.136021 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9b4d67-805a-4456-a23b-3f6ab0420e01 None admin] RBAC: Authorizing `identity:get_project(project_id=8d00e05132b54e9b856788c06cdb7a56)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:31.137465 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9b4d67-805a-4456-a23b-3f6ab0420e01 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:31.139119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 42/85] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:30 2022] GET /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56 => generated 298 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:31.139878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9b4d67-805a-4456-a23b-3f6ab0420e01 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:31.148890 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b87121-153b-4c85-8704-23dc9aa3c196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:31.149527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b87121-153b-4c85-8704-23dc9aa3c196 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:31.149810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b87121-153b-4c85-8704-23dc9aa3c196 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:31.150093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b87121-153b-4c85-8704-23dc9aa3c196 None admin] PATH_INFO: `/v3/users/5eef2e8ac0a34e458d28dc3d163cb041` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:31.151278 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b87121-153b-4c85-8704-23dc9aa3c196 None admin] RBAC: Authorizing `identity:get_user(user_id=5eef2e8ac0a34e458d28dc3d163cb041)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:31.152718 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b87121-153b-4c85-8704-23dc9aa3c196 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:31.154552 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 44/86] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:31 2022] GET /identity/v3/users/5eef2e8ac0a34e458d28dc3d163cb041 => generated 275 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:31.155315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b87121-153b-4c85-8704-23dc9aa3c196 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:31.165783 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:31.180465 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:31.238052 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9b4d67-805a-4456-a23b-3f6ab0420e01 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:31.253713 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b87121-153b-4c85-8704-23dc9aa3c196 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:31.272062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3427f9-5855-454e-afa7-62e405b9858d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:31.272701 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3427f9-5855-454e-afa7-62e405b9858d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:31.272993 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3427f9-5855-454e-afa7-62e405b9858d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:31.273271 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3427f9-5855-454e-afa7-62e405b9858d None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:31.274392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3427f9-5855-454e-afa7-62e405b9858d None admin] RBAC: Authorizing `identity:get_project(project_id=8d00e05132b54e9b856788c06cdb7a56)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:31.275862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3427f9-5855-454e-afa7-62e405b9858d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:31.277656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 43/87] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:31 2022] GET /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56 => generated 298 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:31.279680 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3427f9-5855-454e-afa7-62e405b9858d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:31.292689 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aef08be-cc1d-441d-bc4a-53a994328ac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:31.293315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aef08be-cc1d-441d-bc4a-53a994328ac5 None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:31.293602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aef08be-cc1d-441d-bc4a-53a994328ac5 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:31.293882 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aef08be-cc1d-441d-bc4a-53a994328ac5 None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:31.294543 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aef08be-cc1d-441d-bc4a-53a994328ac5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:31.296059 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aef08be-cc1d-441d-bc4a-53a994328ac5 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:31.377324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3427f9-5855-454e-afa7-62e405b9858d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:31.384299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:31.410968 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1dd961-0757-4c54-ab2c-d10ca75eec3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:31.411609 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1dd961-0757-4c54-ab2c-d10ca75eec3d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:31.411852 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 45/88] 10.222.0.49 () {60 vars in 1132 bytes} [Fri Sep 16 03:26:31 2022] POST /identity/v3/users => generated 289 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:26:31.411943 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1dd961-0757-4c54-ab2c-d10ca75eec3d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:31.412245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1dd961-0757-4c54-ab2c-d10ca75eec3d None admin] PATH_INFO: `/v3/users/5eef2e8ac0a34e458d28dc3d163cb041` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:31.413479 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1dd961-0757-4c54-ab2c-d10ca75eec3d None admin] RBAC: Authorizing `identity:get_user(user_id=5eef2e8ac0a34e458d28dc3d163cb041)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:31.415119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1dd961-0757-4c54-ab2c-d10ca75eec3d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:31.416112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aef08be-cc1d-441d-bc4a-53a994328ac5 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:31.416930 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 44/89] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:31 2022] GET /identity/v3/users/5eef2e8ac0a34e458d28dc3d163cb041 => generated 275 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:31.417623 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1dd961-0757-4c54-ab2c-d10ca75eec3d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:31.492985 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:31.536835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1dd961-0757-4c54-ab2c-d10ca75eec3d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:31.542607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aef08be-cc1d-441d-bc4a-53a994328ac5 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:31.574675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03bdc3f6-93b6-4b8a-a02b-cbc2219a3196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:31.575357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bdc3f6-93b6-4b8a-a02b-cbc2219a3196 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:31.575779 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bdc3f6-93b6-4b8a-a02b-cbc2219a3196 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:31.575918 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bdc3f6-93b6-4b8a-a02b-cbc2219a3196 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:31.584667 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11229ae-6102-41d1-9d07-8d8bd899a4a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:31.585337 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11229ae-6102-41d1-9d07-8d8bd899a4a1 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:31.585630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11229ae-6102-41d1-9d07-8d8bd899a4a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:31.586638 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11229ae-6102-41d1-9d07-8d8bd899a4a1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:31.586638 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11229ae-6102-41d1-9d07-8d8bd899a4a1 None admin] RBAC: Adding query filter params (role.id=589e1f1333a042b7a00c53384a2f5426, scope.project.id=8d00e05132b54e9b856788c06cdb7a56, user.id=5eef2e8ac0a34e458d28dc3d163cb041) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:31.586977 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11229ae-6102-41d1-9d07-8d8bd899a4a1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:31.588447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11229ae-6102-41d1-9d07-8d8bd899a4a1 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:31.594042 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 46/90] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:26:31 2022] GET /identity/v3/role_assignments?user.id=5eef2e8ac0a34e458d28dc3d163cb041&scope.project.id=8d00e05132b54e9b856788c06cdb7a56&role.id=589e1f1333a042b7a00c53384a2f5426 => generated 259 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:31.595014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11229ae-6102-41d1-9d07-8d8bd899a4a1 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:31.628967 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:31.665632 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03bdc3f6-93b6-4b8a-a02b-cbc2219a3196 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:31.667000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03bdc3f6-93b6-4b8a-a02b-cbc2219a3196 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:31.735763 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11229ae-6102-41d1-9d07-8d8bd899a4a1 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:31.777146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:31.788976 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b94adc6-318f-44a4-8553-38953d800c1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:31.789672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b94adc6-318f-44a4-8553-38953d800c1b None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:31.789960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b94adc6-318f-44a4-8553-38953d800c1b None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:31.790245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b94adc6-318f-44a4-8553-38953d800c1b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:31.791106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b94adc6-318f-44a4-8553-38953d800c1b None admin] RBAC: Adding query filter params (role.id=89c4b3a971224d468d33d2dc411230a2, scope.project.id=8d00e05132b54e9b856788c06cdb7a56, user.id=3e71cde4d415471495fcc3441fcdfc54) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:31.791425 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b94adc6-318f-44a4-8553-38953d800c1b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:31.793798 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b94adc6-318f-44a4-8553-38953d800c1b None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:31.802208 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 47/91] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:26:31 2022] GET /identity/v3/role_assignments?user.id=3e71cde4d415471495fcc3441fcdfc54&scope.project.id=8d00e05132b54e9b856788c06cdb7a56&role.id=89c4b3a971224d468d33d2dc411230a2 => generated 259 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:31.803314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b94adc6-318f-44a4-8553-38953d800c1b None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:31.808657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 45/92] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:26:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:26:31.815275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03bdc3f6-93b6-4b8a-a02b-cbc2219a3196 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:31.918007 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:31.947120 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03bdc3f6-93b6-4b8a-a02b-cbc2219a3196 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:31.989263 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b94adc6-318f-44a4-8553-38953d800c1b None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:32.012122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5d2a7c-bd18-49ca-98c2-3a0b8a22008f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:32.012781 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5d2a7c-bd18-49ca-98c2-3a0b8a22008f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:32.013061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5d2a7c-bd18-49ca-98c2-3a0b8a22008f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:32.013337 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5d2a7c-bd18-49ca-98c2-3a0b8a22008f None admin] PATH_INFO: `/v3/projects/6e564d1b6f644f75b62ab2a98008997a` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:32.014495 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5d2a7c-bd18-49ca-98c2-3a0b8a22008f None admin] RBAC: Authorizing `identity:get_project(project_id=6e564d1b6f644f75b62ab2a98008997a)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:32.016020 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5d2a7c-bd18-49ca-98c2-3a0b8a22008f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:32.018570 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 46/93] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:31 2022] GET /identity/v3/projects/6e564d1b6f644f75b62ab2a98008997a => generated 312 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:26:32.024397 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5d2a7c-bd18-49ca-98c2-3a0b8a22008f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:32.073219 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b30e06b-1320-4ffd-9ad3-d57d8f56fb56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:32.073852 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b30e06b-1320-4ffd-9ad3-d57d8f56fb56 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:32.074134 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b30e06b-1320-4ffd-9ad3-d57d8f56fb56 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:32.074409 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b30e06b-1320-4ffd-9ad3-d57d8f56fb56 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:32.197537 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:32.217593 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b30e06b-1320-4ffd-9ad3-d57d8f56fb56 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:32.219782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b30e06b-1320-4ffd-9ad3-d57d8f56fb56 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:32.226932 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5d2a7c-bd18-49ca-98c2-3a0b8a22008f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:32.323234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02b98c4-3f34-40a9-9477-3cee64fff74a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:32.323896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02b98c4-3f34-40a9-9477-3cee64fff74a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:32.324185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02b98c4-3f34-40a9-9477-3cee64fff74a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:32.324465 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02b98c4-3f34-40a9-9477-3cee64fff74a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:32.325214 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02b98c4-3f34-40a9-9477-3cee64fff74a None admin] RBAC: Adding query filter params (role.id=589e1f1333a042b7a00c53384a2f5426, scope.project.id=6e564d1b6f644f75b62ab2a98008997a, user.id=5eef2e8ac0a34e458d28dc3d163cb041) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:32.325525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02b98c4-3f34-40a9-9477-3cee64fff74a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:32.329324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02b98c4-3f34-40a9-9477-3cee64fff74a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:32.344650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 47/94] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:26:32 2022] GET /identity/v3/role_assignments?user.id=5eef2e8ac0a34e458d28dc3d163cb041&scope.project.id=6e564d1b6f644f75b62ab2a98008997a&role.id=589e1f1333a042b7a00c53384a2f5426 => generated 259 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:32.430396 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 48/95] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:26:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 628 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:26:32.436243 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02b98c4-3f34-40a9-9477-3cee64fff74a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:32.447865 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:32.594296 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02b98c4-3f34-40a9-9477-3cee64fff74a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:32.631386 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f3540d-d3b0-434c-bf78-bb3868cdaf59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:32.632282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f3540d-d3b0-434c-bf78-bb3868cdaf59 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:32.632707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f3540d-d3b0-434c-bf78-bb3868cdaf59 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:32.633123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f3540d-d3b0-434c-bf78-bb3868cdaf59 None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:32.634571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f3540d-d3b0-434c-bf78-bb3868cdaf59 None admin] RBAC: Authorizing `identity:get_project(project_id=8d00e05132b54e9b856788c06cdb7a56)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:32.636323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f3540d-d3b0-434c-bf78-bb3868cdaf59 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:32.638204 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 48/96] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:32 2022] GET /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56 => generated 298 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:32.645143 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b30e06b-1320-4ffd-9ad3-d57d8f56fb56 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:32.762004 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:32.771784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b30e06b-1320-4ffd-9ad3-d57d8f56fb56 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:32.817350 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2114eb07-54f7-4728-b0fa-7ce91563dbe2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:32.818027 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2114eb07-54f7-4728-b0fa-7ce91563dbe2 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:32.818316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2114eb07-54f7-4728-b0fa-7ce91563dbe2 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:32.818589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2114eb07-54f7-4728-b0fa-7ce91563dbe2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:32.819353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2114eb07-54f7-4728-b0fa-7ce91563dbe2 None admin] RBAC: Adding query filter params (role.id=fa8bdf568182483aac8a14ef98a12a95, scope.project.id=8d00e05132b54e9b856788c06cdb7a56, user.id=5eef2e8ac0a34e458d28dc3d163cb041) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:32.819661 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2114eb07-54f7-4728-b0fa-7ce91563dbe2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:32.821130 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2114eb07-54f7-4728-b0fa-7ce91563dbe2 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:32.827856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 49/97] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:26:32 2022] GET /identity/v3/role_assignments?user.id=5eef2e8ac0a34e458d28dc3d163cb041&scope.project.id=8d00e05132b54e9b856788c06cdb7a56&role.id=fa8bdf568182483aac8a14ef98a12a95 => generated 259 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:34.547382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d16e5f-0ab7-4c62-9b10-ae0d5d3afe3a None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:34.548480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d16e5f-0ab7-4c62-9b10-ae0d5d3afe3a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:34.549077 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d16e5f-0ab7-4c62-9b10-ae0d5d3afe3a None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:34.550669 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 49/98] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:34.556354 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7163c49c-9078-4bd9-9ac6-833e4e84c454 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:34.557216 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7163c49c-9078-4bd9-9ac6-833e4e84c454 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:34.557711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7163c49c-9078-4bd9-9ac6-833e4e84c454 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:34.581044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-7163c49c-9078-4bd9-9ac6-833e4e84c454 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:34.582518 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-7163c49c-9078-4bd9-9ac6-833e4e84c454 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:34.670342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 50/99] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:34.708358 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b683c0-41ef-4dd5-b60d-0ca4ea1887f7 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:34.708971 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b683c0-41ef-4dd5-b60d-0ca4ea1887f7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:34.709451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b683c0-41ef-4dd5-b60d-0ca4ea1887f7 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:34.711017 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 50/100] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:34.717016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b918bfd9-5afa-4bea-8e0b-334bf49eab61 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:34.717641 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b918bfd9-5afa-4bea-8e0b-334bf49eab61 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:34.718124 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b918bfd9-5afa-4bea-8e0b-334bf49eab61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:34.739824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-b918bfd9-5afa-4bea-8e0b-334bf49eab61 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:34.741324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b918bfd9-5afa-4bea-8e0b-334bf49eab61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:34.744027 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ecb486-ca9c-4909-ba4f-089b50fd0096 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:34.744548 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ecb486-ca9c-4909-ba4f-089b50fd0096 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:34.745023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ecb486-ca9c-4909-ba4f-089b50fd0096 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:34.746545 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 51/101] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:34.751812 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a79fb4-5a67-4b29-a4f3-f37ba5668504 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:34.752510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a79fb4-5a67-4b29-a4f3-f37ba5668504 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:34.752985 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a79fb4-5a67-4b29-a4f3-f37ba5668504 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:34.779625 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-24a79fb4-5a67-4b29-a4f3-f37ba5668504 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:34.782640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-24a79fb4-5a67-4b29-a4f3-f37ba5668504 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:34.873006 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 51/102] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:34.885239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 52/103] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 03:26:35.079400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a327987-4f98-4556-885f-d60c462e2994 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:35.080207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a327987-4f98-4556-885f-d60c462e2994 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:35.080700 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a327987-4f98-4556-885f-d60c462e2994 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:35.082320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 52/104] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:35.087791 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd677bcb-89d7-4057-9f61-32b54813a237 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:35.088350 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd677bcb-89d7-4057-9f61-32b54813a237 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:35.088826 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd677bcb-89d7-4057-9f61-32b54813a237 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:35.100011 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:35.111103 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-fd677bcb-89d7-4057-9f61-32b54813a237 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:35.112296 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fd677bcb-89d7-4057-9f61-32b54813a237 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:35.164400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a327987-4f98-4556-885f-d60c462e2994 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:35.192167 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:35.200537 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 53/105] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:35.245410 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd677bcb-89d7-4057-9f61-32b54813a237 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:35.386951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a327987-4f98-4556-885f-d60c462e2994 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:35.451860 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd677bcb-89d7-4057-9f61-32b54813a237 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:35.463717 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7aac847-ce09-450f-8a58-532acc232820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:35.464362 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7aac847-ce09-450f-8a58-532acc232820 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:35.464674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7aac847-ce09-450f-8a58-532acc232820 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:35.464919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7aac847-ce09-450f-8a58-532acc232820 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:35.483567 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7aac847-ce09-450f-8a58-532acc232820 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:35.489194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-f7aac847-ce09-450f-8a58-532acc232820 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:26:35.489194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:35.489194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:35.489194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:35.489194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:35.489194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:35.489194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:35.489194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:35.489194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:35.489194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:35.489194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:35.489194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:35.489194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:35.489194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:35.489194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:35.489194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:35.490220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:35.490220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:35.490220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:35.490220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:35.490220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:35.490220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:35.490220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:35.490220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:35.490220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:35.490220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:35.490220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:35.490220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:35.490220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:35.490220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:35.490220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:35.490220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:35.491217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:35.491217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:35.491217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:35.491217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:35.491217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:35.491217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:35.491217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:35.491217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:35.491217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:35.491217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:35.491217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:26:35.491217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:35.491217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 53/106] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:35 2022] GET /identity/v3/roles/member => generated 84 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:35.491217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7aac847-ce09-450f-8a58-532acc232820 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:35.527732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a59fba-bf9d-48ce-ad0b-e11f95a3485e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:35.528423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a59fba-bf9d-48ce-ad0b-e11f95a3485e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:35.529064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a59fba-bf9d-48ce-ad0b-e11f95a3485e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:35.529347 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a59fba-bf9d-48ce-ad0b-e11f95a3485e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:35.532145 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a59fba-bf9d-48ce-ad0b-e11f95a3485e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:35.533863 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a59fba-bf9d-48ce-ad0b-e11f95a3485e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:35.535464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 54/107] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:26:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:26:35.538999 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a59fba-bf9d-48ce-ad0b-e11f95a3485e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:35.542345 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:35.603610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:35.628708 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7aac847-ce09-450f-8a58-532acc232820 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:35.664880 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02de59be-661f-4835-8cd1-36da674e12d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:35.665918 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02de59be-661f-4835-8cd1-36da674e12d7 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:35.666415 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02de59be-661f-4835-8cd1-36da674e12d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:35.666900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02de59be-661f-4835-8cd1-36da674e12d7 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:35.667677 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a59fba-bf9d-48ce-ad0b-e11f95a3485e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:35.679007 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02de59be-661f-4835-8cd1-36da674e12d7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:35.684237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-02de59be-661f-4835-8cd1-36da674e12d7 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 03:26:35.684237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:35.684237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:35.684237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:35.684237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:35.684237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:35.684237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:35.684237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:35.684237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:35.684237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:35.684237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:35.684237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:35.684237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:35.684237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:35.684237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:35.684237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:35.685086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:35.685086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:35.685086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:35.685086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:35.685086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:35.685086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:35.685086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:35.685086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:35.685086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:35.685086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:35.685086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:35.685086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:35.685086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:35.685086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:35.685086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:35.685086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:35.685869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:35.685869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:35.685869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:35.685869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:35.685869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:35.685869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:35.685869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:35.685869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:35.685869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:35.685869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:35.685869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 03:26:35.685869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:35.693506 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 54/108] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:35 2022] GET /identity/v3/roles/reader => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:35.694565 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02de59be-661f-4835-8cd1-36da674e12d7 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:35.705086 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7419b7c0-47e8-409d-8f5f-c17918259f90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:35.705555 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7419b7c0-47e8-409d-8f5f-c17918259f90 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:35.705903 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7419b7c0-47e8-409d-8f5f-c17918259f90 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:35.706186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7419b7c0-47e8-409d-8f5f-c17918259f90 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:35.706957 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7419b7c0-47e8-409d-8f5f-c17918259f90 None admin] RBAC: Adding query filter params (name=member) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:35.707460 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7419b7c0-47e8-409d-8f5f-c17918259f90 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:35.709273 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7419b7c0-47e8-409d-8f5f-c17918259f90 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:35.719999 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 55/109] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:26:35 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:35.731329 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7419b7c0-47e8-409d-8f5f-c17918259f90 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:35.791721 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:35.811674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02de59be-661f-4835-8cd1-36da674e12d7 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:35.831340 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7419b7c0-47e8-409d-8f5f-c17918259f90 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:35.849140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd30a957-650c-4739-ae00-a70a5643a040 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:35.850035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd30a957-650c-4739-ae00-a70a5643a040 None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:35.850492 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd30a957-650c-4739-ae00-a70a5643a040 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:35.851106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd30a957-650c-4739-ae00-a70a5643a040 None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:35.852034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd30a957-650c-4739-ae00-a70a5643a040 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:35.853770 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd30a957-650c-4739-ae00-a70a5643a040 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:35.859925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:35.870590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2aab06-6d22-4063-9e79-9d072efd3629 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:35.871340 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2aab06-6d22-4063-9e79-9d072efd3629 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:35.871696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2aab06-6d22-4063-9e79-9d072efd3629 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:35.872091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2aab06-6d22-4063-9e79-9d072efd3629 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:35.883409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2aab06-6d22-4063-9e79-9d072efd3629 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:35.888390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-2c2aab06-6d22-4063-9e79-9d072efd3629 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:26:35.888390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:35.888390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:35.888390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:35.888390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:35.888390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:35.888390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:35.888390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:35.888390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:35.888390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:35.888390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:35.888390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:35.888390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:35.888390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:35.888390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:35.888390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:35.892403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:35.892403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:35.892403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:35.892403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:35.892403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:35.892403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:35.892403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:35.892403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:35.892403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:35.892403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:35.892403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:35.892403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:35.892403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:35.892403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:35.892403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:35.892403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:35.893196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:35.893196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:35.893196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:35.893196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:35.893196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:35.893196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:35.893196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:35.893196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:35.893196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:35.893196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:35.893196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:26:35.893196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:26:35.893196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 56/110] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:26:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:35.893196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2aab06-6d22-4063-9e79-9d072efd3629 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:35.948154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 55/111] 10.222.0.49 () {60 vars in 1141 bytes} [Fri Sep 16 03:26:35 2022] POST /identity/v3/projects => generated 302 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:26:35.949784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd30a957-650c-4739-ae00-a70a5643a040 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:36.024964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2aab06-6d22-4063-9e79-9d072efd3629 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:36.063454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:36.087420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b4f101-c5cf-4f92-995a-55ae0e46d016 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:36.088115 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b4f101-c5cf-4f92-995a-55ae0e46d016 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:36.088404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b4f101-c5cf-4f92-995a-55ae0e46d016 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:36.088805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b4f101-c5cf-4f92-995a-55ae0e46d016 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:36.089375 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b4f101-c5cf-4f92-995a-55ae0e46d016 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:36.089685 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b4f101-c5cf-4f92-995a-55ae0e46d016 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:36.091195 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b4f101-c5cf-4f92-995a-55ae0e46d016 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:36.095091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd30a957-650c-4739-ae00-a70a5643a040 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:36.110965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 57/112] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:26:35 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:36.110965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b4f101-c5cf-4f92-995a-55ae0e46d016 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:36.146472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760d98ac-fafe-4493-9845-cd2a708b3d87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:36.147328 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760d98ac-fafe-4493-9845-cd2a708b3d87 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:36.147867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760d98ac-fafe-4493-9845-cd2a708b3d87 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:36.148146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760d98ac-fafe-4493-9845-cd2a708b3d87 None admin] PATH_INFO: `/v3/users/5eef2e8ac0a34e458d28dc3d163cb041` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:36.149348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760d98ac-fafe-4493-9845-cd2a708b3d87 None admin] RBAC: Authorizing `identity:get_user(user_id=5eef2e8ac0a34e458d28dc3d163cb041)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:36.151348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760d98ac-fafe-4493-9845-cd2a708b3d87 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:36.159117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 56/113] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:35 2022] GET /identity/v3/users/5eef2e8ac0a34e458d28dc3d163cb041 => generated 275 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:36.160171 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760d98ac-fafe-4493-9845-cd2a708b3d87 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:36.239153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:36.285543 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b4f101-c5cf-4f92-995a-55ae0e46d016 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:36.296314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760d98ac-fafe-4493-9845-cd2a708b3d87 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:36.330635 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8adeb9-943f-4f3a-a1db-54bc4e4e2560 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:36.331468 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8adeb9-943f-4f3a-a1db-54bc4e4e2560 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:36.331756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8adeb9-943f-4f3a-a1db-54bc4e4e2560 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:36.332098 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8adeb9-943f-4f3a-a1db-54bc4e4e2560 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:36.332778 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8adeb9-943f-4f3a-a1db-54bc4e4e2560 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:36.333083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8adeb9-943f-4f3a-a1db-54bc4e4e2560 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:36.334462 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8adeb9-943f-4f3a-a1db-54bc4e4e2560 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:36.339255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d2646f-5d6f-4da3-968b-d287cb429602 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:36.339904 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d2646f-5d6f-4da3-968b-d287cb429602 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:36.340183 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d2646f-5d6f-4da3-968b-d287cb429602 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:36.340459 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d2646f-5d6f-4da3-968b-d287cb429602 None admin] PATH_INFO: `/v3/users/af186a335cba418dbdfe79d5fb7a055c` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:36.347363 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 58/114] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:36.348210 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8adeb9-943f-4f3a-a1db-54bc4e4e2560 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:36.356637 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d2646f-5d6f-4da3-968b-d287cb429602 None admin] RBAC: Authorizing `identity:get_user(user_id=af186a335cba418dbdfe79d5fb7a055c)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:36.358527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d2646f-5d6f-4da3-968b-d287cb429602 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:36.360614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 57/115] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:36 2022] GET /identity/v3/users/af186a335cba418dbdfe79d5fb7a055c => generated 289 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:36.362493 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d9d7c0-21ed-459c-8357-5b8c0d98283a None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:36.362693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d9d7c0-21ed-459c-8357-5b8c0d98283a None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:36.366956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d9d7c0-21ed-459c-8357-5b8c0d98283a None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:36.366956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 58/116] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:36.366956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e720e8e7-34f9-4fb4-b456-b71a0c9b5e70 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:36.366956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e720e8e7-34f9-4fb4-b456-b71a0c9b5e70 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:36.366956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e720e8e7-34f9-4fb4-b456-b71a0c9b5e70 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:36.367573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 59/117] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:36.368243 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e720e8e7-34f9-4fb4-b456-b71a0c9b5e70 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:36.412332 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:36.466549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8adeb9-943f-4f3a-a1db-54bc4e4e2560 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:36.476678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e720e8e7-34f9-4fb4-b456-b71a0c9b5e70 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:36.491878 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:36.506226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6716e0-fb52-43fe-a6c6-415a448b8b9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:36.506878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6716e0-fb52-43fe-a6c6-415a448b8b9f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:36.507193 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6716e0-fb52-43fe-a6c6-415a448b8b9f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:36.507473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6716e0-fb52-43fe-a6c6-415a448b8b9f None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:36.508597 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6716e0-fb52-43fe-a6c6-415a448b8b9f None admin] RBAC: Authorizing `identity:get_project(project_id=8d00e05132b54e9b856788c06cdb7a56)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:36.510092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6716e0-fb52-43fe-a6c6-415a448b8b9f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:36.511763 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 59/118] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:36 2022] GET /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56 => generated 298 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:36.516527 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6716e0-fb52-43fe-a6c6-415a448b8b9f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:36.519006 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29c76b2-2ac6-4608-b867-58db49f300e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:36.519885 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29c76b2-2ac6-4608-b867-58db49f300e4 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:36.520307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29c76b2-2ac6-4608-b867-58db49f300e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:36.520720 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29c76b2-2ac6-4608-b867-58db49f300e4 None admin] PATH_INFO: `/v3/users/3e71cde4d415471495fcc3441fcdfc54` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:36.521984 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29c76b2-2ac6-4608-b867-58db49f300e4 None admin] RBAC: Authorizing `identity:get_user(user_id=3e71cde4d415471495fcc3441fcdfc54)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:36.523471 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29c76b2-2ac6-4608-b867-58db49f300e4 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:36.530199 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 60/119] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:36 2022] GET /identity/v3/users/3e71cde4d415471495fcc3441fcdfc54 => generated 247 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:36.531379 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3c1154-c3fd-4ca2-a4b1-4ca1c2467f1c None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:36.531780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3c1154-c3fd-4ca2-a4b1-4ca1c2467f1c None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:36.532079 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3c1154-c3fd-4ca2-a4b1-4ca1c2467f1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:36.551102 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-9a3c1154-c3fd-4ca2-a4b1-4ca1c2467f1c None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:36.552223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-9a3c1154-c3fd-4ca2-a4b1-4ca1c2467f1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:36.620315 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6716e0-fb52-43fe-a6c6-415a448b8b9f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:36.628456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 61/120] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:36.628456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f5a49f-6154-47fe-9dc7-ee6086a13130 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:36.628456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f5a49f-6154-47fe-9dc7-ee6086a13130 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:36.628456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f5a49f-6154-47fe-9dc7-ee6086a13130 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:36.645288 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-c2f5a49f-6154-47fe-9dc7-ee6086a13130 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:36.646536 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c2f5a49f-6154-47fe-9dc7-ee6086a13130 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:36.651039 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:36.670822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667b1769-4c03-4d20-8a44-945c001f4d85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:36.671823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667b1769-4c03-4d20-8a44-945c001f4d85 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:36.672202 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667b1769-4c03-4d20-8a44-945c001f4d85 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:36.672652 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667b1769-4c03-4d20-8a44-945c001f4d85 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:36.717427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 62/121] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:36.718260 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f5a49f-6154-47fe-9dc7-ee6086a13130 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:36.753152 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667b1769-4c03-4d20-8a44-945c001f4d85 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:36.755004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667b1769-4c03-4d20-8a44-945c001f4d85 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:36.763118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:36.835249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f5a49f-6154-47fe-9dc7-ee6086a13130 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:36.877547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 60/122] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:26:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:26:36.878345 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667b1769-4c03-4d20-8a44-945c001f4d85 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:36.905111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb9de5b-35d1-4215-8c5a-3b152cdf4ac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:36.905799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb9de5b-35d1-4215-8c5a-3b152cdf4ac5 None admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:36.906084 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb9de5b-35d1-4215-8c5a-3b152cdf4ac5 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:36.906361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb9de5b-35d1-4215-8c5a-3b152cdf4ac5 None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56/users/5eef2e8ac0a34e458d28dc3d163cb041/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:36.911436 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb9de5b-35d1-4215-8c5a-3b152cdf4ac5 None admin] RBAC: Authorizing `identity:create_grant(project_id=8d00e05132b54e9b856788c06cdb7a56, user_id=5eef2e8ac0a34e458d28dc3d163cb041, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:36.912916 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb9de5b-35d1-4215-8c5a-3b152cdf4ac5 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:36.916705 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:36.998148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 63/123] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:26:36 2022] PUT /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56/users/5eef2e8ac0a34e458d28dc3d163cb041/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:26:37.003149 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb9de5b-35d1-4215-8c5a-3b152cdf4ac5 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:37.016297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667b1769-4c03-4d20-8a44-945c001f4d85 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:37.062554 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d58585-cf5b-464b-9be1-2f136b3dacb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:37.063721 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d58585-cf5b-464b-9be1-2f136b3dacb5 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:37.064292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d58585-cf5b-464b-9be1-2f136b3dacb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:37.064797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d58585-cf5b-464b-9be1-2f136b3dacb5 None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:37.067631 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d58585-cf5b-464b-9be1-2f136b3dacb5 None admin] RBAC: Authorizing `identity:get_project(project_id=8d00e05132b54e9b856788c06cdb7a56)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:37.069551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d58585-cf5b-464b-9be1-2f136b3dacb5 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:37.071614 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 61/124] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:36 2022] GET /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56 => generated 298 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:37.072291 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d58585-cf5b-464b-9be1-2f136b3dacb5 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:37.154779 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb9de5b-35d1-4215-8c5a-3b152cdf4ac5 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:37.157782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:37.191866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:37.206568 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd35598-4a4e-4fc4-a23a-305622f15708 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:37.207260 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd35598-4a4e-4fc4-a23a-305622f15708 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:37.207547 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd35598-4a4e-4fc4-a23a-305622f15708 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:37.207831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd35598-4a4e-4fc4-a23a-305622f15708 None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:37.209740 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd35598-4a4e-4fc4-a23a-305622f15708 None admin] RBAC: Authorizing `identity:get_project(project_id=8d00e05132b54e9b856788c06cdb7a56)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:37.211324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd35598-4a4e-4fc4-a23a-305622f15708 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:37.213447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 64/125] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:36 2022] GET /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56 => generated 298 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:37.219146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd35598-4a4e-4fc4-a23a-305622f15708 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:37.221918 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d58585-cf5b-464b-9be1-2f136b3dacb5 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:37.261432 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a7fe29-1f88-4eb5-9f76-3d2503a82fed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:37.262331 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a7fe29-1f88-4eb5-9f76-3d2503a82fed None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:37.262755 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a7fe29-1f88-4eb5-9f76-3d2503a82fed None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:37.263197 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a7fe29-1f88-4eb5-9f76-3d2503a82fed None admin] PATH_INFO: `/v3/roles/member` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:37.279494 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a7fe29-1f88-4eb5-9f76-3d2503a82fed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:37.285580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-51a7fe29-1f88-4eb5-9f76-3d2503a82fed None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:26:37.285580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:37.285580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:37.285580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:37.285580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:37.285580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:37.285580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:37.285580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:37.285580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:37.285580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:37.285580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:37.285580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:37.285580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:37.285580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:37.285580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:37.285580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:37.286657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:37.286657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:37.286657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:37.286657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:37.286657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:37.286657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:37.286657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:37.286657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:37.286657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:37.286657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:37.286657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:37.286657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:37.286657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:37.286657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:37.286657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:37.286657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:37.287526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:37.287526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:37.287526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:37.287526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:37.287526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:37.287526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:37.287526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:37.287526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:37.287526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:37.287526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:37.287526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:26:37.287526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:26:37.289305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 62/126] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:37.290241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a7fe29-1f88-4eb5-9f76-3d2503a82fed None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:37.348850 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd35598-4a4e-4fc4-a23a-305622f15708 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:37.384794 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcbf6654-5f4b-41fd-a8fb-5c702f9d3017 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:37.385804 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcbf6654-5f4b-41fd-a8fb-5c702f9d3017 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:37.386310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcbf6654-5f4b-41fd-a8fb-5c702f9d3017 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:37.386825 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcbf6654-5f4b-41fd-a8fb-5c702f9d3017 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:37.398064 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcbf6654-5f4b-41fd-a8fb-5c702f9d3017 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:37.404222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-bcbf6654-5f4b-41fd-a8fb-5c702f9d3017 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:26:37.404222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:37.404222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:37.404222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:37.404222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:37.404222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:37.404222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:37.404222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:37.404222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:37.404222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:37.404222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:37.404222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:37.404222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:37.404222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:37.404222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:37.404222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:37.405162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:37.405162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:37.405162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:37.405162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:37.405162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:37.405162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:37.405162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:37.405162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:37.405162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:37.405162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:37.405162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:37.405162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:37.405162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:37.405162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:37.405162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:37.405162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:37.405943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:37.405943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:37.405943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:37.405943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:37.405943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:37.405943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:37.405943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:37.405943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:37.405943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:37.405943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:37.405943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:26:37.405943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:37.408919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 65/127] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:26:37 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:37.411117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcbf6654-5f4b-41fd-a8fb-5c702f9d3017 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:37.420555 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a7fe29-1f88-4eb5-9f76-3d2503a82fed None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:37.454252 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:37.466310 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cd152c-d125-4457-a2c9-2cd0126844dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:37.467898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cd152c-d125-4457-a2c9-2cd0126844dc None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:37.467898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cd152c-d125-4457-a2c9-2cd0126844dc None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:37.467898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cd152c-d125-4457-a2c9-2cd0126844dc None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56/users/3e71cde4d415471495fcc3441fcdfc54/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:37.472814 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cd152c-d125-4457-a2c9-2cd0126844dc None admin] RBAC: Authorizing `identity:create_grant(project_id=8d00e05132b54e9b856788c06cdb7a56, user_id=3e71cde4d415471495fcc3441fcdfc54, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:37.475187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cd152c-d125-4457-a2c9-2cd0126844dc None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:37.535245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcbf6654-5f4b-41fd-a8fb-5c702f9d3017 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:37.572985 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac8c572-2cf5-47d8-b5ac-521f679081f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:37.573332 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 63/128] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:26:37 2022] PUT /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56/users/3e71cde4d415471495fcc3441fcdfc54/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:26:37.573609 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac8c572-2cf5-47d8-b5ac-521f679081f8 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:37.573893 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac8c572-2cf5-47d8-b5ac-521f679081f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:37.574171 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac8c572-2cf5-47d8-b5ac-521f679081f8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:37.574976 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac8c572-2cf5-47d8-b5ac-521f679081f8 None admin] RBAC: Adding query filter params (role.id=363c2654ab4e456b8b9b0b8c8c28911e, scope.project.id=8d00e05132b54e9b856788c06cdb7a56, user.id=af186a335cba418dbdfe79d5fb7a055c) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:37.575293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac8c572-2cf5-47d8-b5ac-521f679081f8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:37.575603 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cd152c-d125-4457-a2c9-2cd0126844dc None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:37.576785 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac8c572-2cf5-47d8-b5ac-521f679081f8 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:37.581730 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:37.592911 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 66/129] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:26:37 2022] GET /identity/v3/role_assignments?user.id=af186a335cba418dbdfe79d5fb7a055c&scope.project.id=8d00e05132b54e9b856788c06cdb7a56&role.id=363c2654ab4e456b8b9b0b8c8c28911e => generated 259 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:37.598939 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac8c572-2cf5-47d8-b5ac-521f679081f8 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:37.733986 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cd152c-d125-4457-a2c9-2cd0126844dc None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:37.777609 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214b7d2a-1e00-4b25-ac4a-4f83cbb153f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:37.779091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214b7d2a-1e00-4b25-ac4a-4f83cbb153f9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:37.779856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214b7d2a-1e00-4b25-ac4a-4f83cbb153f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:37.780914 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214b7d2a-1e00-4b25-ac4a-4f83cbb153f9 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:37.782060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214b7d2a-1e00-4b25-ac4a-4f83cbb153f9 None admin] RBAC: Adding query filter params (name=member) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:37.782821 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214b7d2a-1e00-4b25-ac4a-4f83cbb153f9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:37.784903 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214b7d2a-1e00-4b25-ac4a-4f83cbb153f9 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:37.793980 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:37.797792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 64/130] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:26:37 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:37.804339 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac8c572-2cf5-47d8-b5ac-521f679081f8 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:37.805097 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214b7d2a-1e00-4b25-ac4a-4f83cbb153f9 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:37.846668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57859b1d-48b0-4154-bf3d-66f00e816f99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:37.847322 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57859b1d-48b0-4154-bf3d-66f00e816f99 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:37.847609 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57859b1d-48b0-4154-bf3d-66f00e816f99 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:37.847889 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57859b1d-48b0-4154-bf3d-66f00e816f99 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:37.848563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57859b1d-48b0-4154-bf3d-66f00e816f99 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:37.848871 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57859b1d-48b0-4154-bf3d-66f00e816f99 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:37.850511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57859b1d-48b0-4154-bf3d-66f00e816f99 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:37.862264 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 67/131] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:26:37 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:37.867728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57859b1d-48b0-4154-bf3d-66f00e816f99 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:37.896609 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:37.951092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214b7d2a-1e00-4b25-ac4a-4f83cbb153f9 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:37.986634 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57859b1d-48b0-4154-bf3d-66f00e816f99 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:37.999798 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2558110d-e3ca-4102-a191-0f79116be44b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:38.001189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2558110d-e3ca-4102-a191-0f79116be44b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:38.002065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2558110d-e3ca-4102-a191-0f79116be44b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:38.003166 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2558110d-e3ca-4102-a191-0f79116be44b None admin] PATH_INFO: `/v3/users/5eef2e8ac0a34e458d28dc3d163cb041` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:38.005055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2558110d-e3ca-4102-a191-0f79116be44b None admin] RBAC: Authorizing `identity:get_user(user_id=5eef2e8ac0a34e458d28dc3d163cb041)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:38.007066 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2558110d-e3ca-4102-a191-0f79116be44b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:38.009009 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 65/132] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:37 2022] GET /identity/v3/users/5eef2e8ac0a34e458d28dc3d163cb041 => generated 275 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:26:38.015176 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2558110d-e3ca-4102-a191-0f79116be44b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:38.026155 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67290d89-735d-410f-9e9a-4a38315f8953 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:38.027076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67290d89-735d-410f-9e9a-4a38315f8953 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:38.027364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67290d89-735d-410f-9e9a-4a38315f8953 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:38.027648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67290d89-735d-410f-9e9a-4a38315f8953 None admin] PATH_INFO: `/v3/users/5eef2e8ac0a34e458d28dc3d163cb041` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:38.029053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67290d89-735d-410f-9e9a-4a38315f8953 None admin] RBAC: Authorizing `identity:get_user(user_id=5eef2e8ac0a34e458d28dc3d163cb041)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:38.030627 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67290d89-735d-410f-9e9a-4a38315f8953 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:38.032548 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 68/133] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:37 2022] GET /identity/v3/users/5eef2e8ac0a34e458d28dc3d163cb041 => generated 275 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:26:38.038755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67290d89-735d-410f-9e9a-4a38315f8953 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:38.067488 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:38.169847 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2558110d-e3ca-4102-a191-0f79116be44b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:38.180125 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67290d89-735d-410f-9e9a-4a38315f8953 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:38.216894 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ade088b-a3b9-40cb-884c-b1c219cd28b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:38.217799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ade088b-a3b9-40cb-884c-b1c219cd28b0 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:38.218222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ade088b-a3b9-40cb-884c-b1c219cd28b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:38.218634 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ade088b-a3b9-40cb-884c-b1c219cd28b0 None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:38.223509 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2070d42f-807d-4b40-b636-1e4b51ac0a8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:38.223840 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ade088b-a3b9-40cb-884c-b1c219cd28b0 None admin] RBAC: Authorizing `identity:get_project(project_id=8d00e05132b54e9b856788c06cdb7a56)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:38.224156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2070d42f-807d-4b40-b636-1e4b51ac0a8d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:38.224440 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2070d42f-807d-4b40-b636-1e4b51ac0a8d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:38.224719 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2070d42f-807d-4b40-b636-1e4b51ac0a8d None admin] PATH_INFO: `/v3/projects/6e564d1b6f644f75b62ab2a98008997a` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:38.225863 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2070d42f-807d-4b40-b636-1e4b51ac0a8d None admin] RBAC: Authorizing `identity:get_project(project_id=6e564d1b6f644f75b62ab2a98008997a)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:38.226085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ade088b-a3b9-40cb-884c-b1c219cd28b0 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:38.227319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2070d42f-807d-4b40-b636-1e4b51ac0a8d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:38.234963 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 69/134] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:38 2022] GET /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56 => generated 298 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:38.235122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 66/135] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:38 2022] GET /identity/v3/projects/6e564d1b6f644f75b62ab2a98008997a => generated 312 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:26:38.236217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2070d42f-807d-4b40-b636-1e4b51ac0a8d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:38.237235 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ade088b-a3b9-40cb-884c-b1c219cd28b0 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:38.243209 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:38.399727 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:38.410458 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ade088b-a3b9-40cb-884c-b1c219cd28b0 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:38.450155 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2070d42f-807d-4b40-b636-1e4b51ac0a8d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:38.475449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45455ccf-ce5f-407c-bb5e-a26ff0c117dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:38.476130 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45455ccf-ce5f-407c-bb5e-a26ff0c117dc None admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:38.476414 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45455ccf-ce5f-407c-bb5e-a26ff0c117dc None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:38.476701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45455ccf-ce5f-407c-bb5e-a26ff0c117dc None admin] PATH_INFO: `/v3/projects/6e564d1b6f644f75b62ab2a98008997a/users/5eef2e8ac0a34e458d28dc3d163cb041/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:38.483220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45455ccf-ce5f-407c-bb5e-a26ff0c117dc None admin] RBAC: Authorizing `identity:create_grant(project_id=6e564d1b6f644f75b62ab2a98008997a, user_id=5eef2e8ac0a34e458d28dc3d163cb041, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:38.484686 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45455ccf-ce5f-407c-bb5e-a26ff0c117dc None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:38.512758 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405bdab3-0aa0-4115-aab3-047d6b7f8d1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:38.513406 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405bdab3-0aa0-4115-aab3-047d6b7f8d1f None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:38.513742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405bdab3-0aa0-4115-aab3-047d6b7f8d1f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:38.514029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405bdab3-0aa0-4115-aab3-047d6b7f8d1f None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56/users/5eef2e8ac0a34e458d28dc3d163cb041/roles/fa8bdf568182483aac8a14ef98a12a95` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:38.519766 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405bdab3-0aa0-4115-aab3-047d6b7f8d1f None admin] RBAC: Authorizing `identity:create_grant(project_id=8d00e05132b54e9b856788c06cdb7a56, user_id=5eef2e8ac0a34e458d28dc3d163cb041, role_id=fa8bdf568182483aac8a14ef98a12a95)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:38.521323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405bdab3-0aa0-4115-aab3-047d6b7f8d1f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:38.555386 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 70/136] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:26:38 2022] PUT /identity/v3/projects/6e564d1b6f644f75b62ab2a98008997a/users/5eef2e8ac0a34e458d28dc3d163cb041/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:26:38.606959 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 67/137] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:26:38 2022] PUT /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56/users/5eef2e8ac0a34e458d28dc3d163cb041/roles/fa8bdf568182483aac8a14ef98a12a95 => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:26:39.105186 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f9162a-de04-4762-9d7b-89fb5228d66b None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:39.105712 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f9162a-de04-4762-9d7b-89fb5228d66b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:39.105712 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f9162a-de04-4762-9d7b-89fb5228d66b None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:39.107092 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 71/138] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:39.112936 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49821b9d-76ce-4375-a666-f54a836e532d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:39.113353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49821b9d-76ce-4375-a666-f54a836e532d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:39.113536 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49821b9d-76ce-4375-a666-f54a836e532d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:39.138282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-49821b9d-76ce-4375-a666-f54a836e532d None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:39.139515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-49821b9d-76ce-4375-a666-f54a836e532d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:39.210840 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:39.221636 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 68/139] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 03:26:39.698406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f9162a-de04-4762-9d7b-89fb5228d66b None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:39.836780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f9162a-de04-4762-9d7b-89fb5228d66b None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:39.879839 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf92d71-ba4f-4c36-b490-7767755001c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:39.880486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf92d71-ba4f-4c36-b490-7767755001c8 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:39.880770 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf92d71-ba4f-4c36-b490-7767755001c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:39.881047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf92d71-ba4f-4c36-b490-7767755001c8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:39.887589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf92d71-ba4f-4c36-b490-7767755001c8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:39.889111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf92d71-ba4f-4c36-b490-7767755001c8 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:39.891005 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 72/140] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:26:39 2022] GET /identity/v3/domains/default => generated 199 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:39.897090 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49821b9d-76ce-4375-a666-f54a836e532d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:39.945118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f625dca-26df-4fa7-be7b-8ef59c168921 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:39.945460 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f625dca-26df-4fa7-be7b-8ef59c168921 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:39.945681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f625dca-26df-4fa7-be7b-8ef59c168921 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:39.947360 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 73/141] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:39.956061 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e7d6c2-c4fa-487a-84db-0d00a4784c4c None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:39.956327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e7d6c2-c4fa-487a-84db-0d00a4784c4c None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:39.956524 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e7d6c2-c4fa-487a-84db-0d00a4784c4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:39.987797 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-47e7d6c2-c4fa-487a-84db-0d00a4784c4c None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:39.989000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-47e7d6c2-c4fa-487a-84db-0d00a4784c4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:40.018144 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49821b9d-76ce-4375-a666-f54a836e532d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:40.019268 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:40.065229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bf23b1-6400-4def-8190-69d955ea900d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:40.065859 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bf23b1-6400-4def-8190-69d955ea900d None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:40.066141 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bf23b1-6400-4def-8190-69d955ea900d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:40.066416 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bf23b1-6400-4def-8190-69d955ea900d None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:40.067126 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bf23b1-6400-4def-8190-69d955ea900d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:40.068574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bf23b1-6400-4def-8190-69d955ea900d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:40.133606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 74/142] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:40.211740 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 69/143] 10.222.0.49 () {60 vars in 1132 bytes} [Fri Sep 16 03:26:39 2022] POST /identity/v3/users => generated 283 bytes in 315 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:26:40.608398 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e7d6c2-c4fa-487a-84db-0d00a4784c4c None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:40.740039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e7d6c2-c4fa-487a-84db-0d00a4784c4c None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:40.797193 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43999b23-0c28-43a9-9153-8bb36e6ad691 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:40.797884 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43999b23-0c28-43a9-9153-8bb36e6ad691 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:40.798824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43999b23-0c28-43a9-9153-8bb36e6ad691 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:40.799114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43999b23-0c28-43a9-9153-8bb36e6ad691 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:40.821226 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43999b23-0c28-43a9-9153-8bb36e6ad691 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:40.826315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-43999b23-0c28-43a9-9153-8bb36e6ad691 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:26:40.826315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:40.826315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:40.826315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:40.826315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:40.826315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:40.826315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:40.826315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:40.826315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:40.826315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:40.826315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:40.826315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:40.826315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:40.826315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:40.826315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:40.826315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:40.828406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:40.828406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:40.828406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:40.828406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:40.828406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:40.828406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:40.828406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:40.828406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:40.828406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:40.828406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:40.828406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:40.828406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:40.828406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:40.828406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:40.828406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:40.828406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:40.830504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:40.830504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:40.830504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:40.830504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:40.830504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:40.830504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:40.830504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:40.830504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:40.830504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:40.830504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:40.830504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:26:40.830504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:40.830504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 75/144] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:40 2022] GET /identity/v3/roles/member => generated 84 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:40.837876 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bf23b1-6400-4def-8190-69d955ea900d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:40.905685 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:40.912064 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545c94d8-ce36-49a4-97b1-92ef56a36a65 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:40.912356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545c94d8-ce36-49a4-97b1-92ef56a36a65 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:40.912641 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545c94d8-ce36-49a4-97b1-92ef56a36a65 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:40.913993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 76/145] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:40.928198 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f939fd5-9950-4dd4-9011-a61d0c34f99c None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:40.928466 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f939fd5-9950-4dd4-9011-a61d0c34f99c None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:40.928753 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f939fd5-9950-4dd4-9011-a61d0c34f99c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:40.960351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-3f939fd5-9950-4dd4-9011-a61d0c34f99c None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:40.961542 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3f939fd5-9950-4dd4-9011-a61d0c34f99c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:40.966848 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:40.968692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bf23b1-6400-4def-8190-69d955ea900d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:41.016499 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fd556c-df21-4d40-a9f2-7798ce0cb26a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:41.017483 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fd556c-df21-4d40-a9f2-7798ce0cb26a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:41.017984 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fd556c-df21-4d40-a9f2-7798ce0cb26a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:41.018539 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fd556c-df21-4d40-a9f2-7798ce0cb26a None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:41.019417 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fd556c-df21-4d40-a9f2-7798ce0cb26a None admin] RBAC: Adding query filter params (name=member) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:41.019899 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fd556c-df21-4d40-a9f2-7798ce0cb26a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:41.021705 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fd556c-df21-4d40-a9f2-7798ce0cb26a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:41.036611 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 70/146] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:26:40 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:41.038119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fdf237-a871-4c86-abdb-022463548a97 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:41.038500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fdf237-a871-4c86-abdb-022463548a97 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:41.039118 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fdf237-a871-4c86-abdb-022463548a97 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:41.040797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 71/147] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:41.048580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a56027-a3ae-4376-a2bc-52a56813edda None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:41.049146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a56027-a3ae-4376-a2bc-52a56813edda None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:41.049688 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a56027-a3ae-4376-a2bc-52a56813edda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:41.061623 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 77/148] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 03:26:41.063036 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f939fd5-9950-4dd4-9011-a61d0c34f99c None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:41.077626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-41a56027-a3ae-4376-a2bc-52a56813edda None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:41.079082 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-41a56027-a3ae-4376-a2bc-52a56813edda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:41.219960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 72/149] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:41.227823 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f939fd5-9950-4dd4-9011-a61d0c34f99c None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:41.266344 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e536e2fc-a5ca-4230-8a04-f0b68e911ca1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:41.267356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e536e2fc-a5ca-4230-8a04-f0b68e911ca1 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:41.267821 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e536e2fc-a5ca-4230-8a04-f0b68e911ca1 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:41.268246 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e536e2fc-a5ca-4230-8a04-f0b68e911ca1 None admin] PATH_INFO: `/v3/users/5eef2e8ac0a34e458d28dc3d163cb041` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:41.271160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e536e2fc-a5ca-4230-8a04-f0b68e911ca1 None admin] RBAC: Authorizing `identity:get_user(user_id=5eef2e8ac0a34e458d28dc3d163cb041)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:41.272797 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e536e2fc-a5ca-4230-8a04-f0b68e911ca1 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:41.274575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 78/150] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:41 2022] GET /identity/v3/users/5eef2e8ac0a34e458d28dc3d163cb041 => generated 275 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:41.280662 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a56027-a3ae-4376-a2bc-52a56813edda None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:41.362651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:41.428723 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a56027-a3ae-4376-a2bc-52a56813edda None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:41.464501 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2c1e35-bd07-4972-9512-f96099c37d04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:41.465127 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2c1e35-bd07-4972-9512-f96099c37d04 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:41.465361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e536e2fc-a5ca-4230-8a04-f0b68e911ca1 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:41.465469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2c1e35-bd07-4972-9512-f96099c37d04 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:41.465669 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2c1e35-bd07-4972-9512-f96099c37d04 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:41.497577 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:41.612381 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2c1e35-bd07-4972-9512-f96099c37d04 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:41.614534 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2c1e35-bd07-4972-9512-f96099c37d04 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:41.618770 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e536e2fc-a5ca-4230-8a04-f0b68e911ca1 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:41.683291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646e4b99-05c3-4aba-9824-c7b08c7e9559 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:41.684196 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646e4b99-05c3-4aba-9824-c7b08c7e9559 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:41.684625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646e4b99-05c3-4aba-9824-c7b08c7e9559 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:41.685075 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646e4b99-05c3-4aba-9824-c7b08c7e9559 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:41.689689 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:41.703258 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646e4b99-05c3-4aba-9824-c7b08c7e9559 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:41.709239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-646e4b99-05c3-4aba-9824-c7b08c7e9559 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 03:26:41.709239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:41.709239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:41.709239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:41.709239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:41.709239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:41.709239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:41.709239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:41.709239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:41.709239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:41.709239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:41.709239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:41.709239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:41.709239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:41.709239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:41.709239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:41.710316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:41.710316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:41.710316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:41.710316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:41.710316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:41.710316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:41.710316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:41.710316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:41.710316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:41.710316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:41.710316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:41.710316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:41.710316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:41.710316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:41.710316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:41.710316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:41.711302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:41.711302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:41.711302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:41.711302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:41.711302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:41.711302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:41.711302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:41.711302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:41.711302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:41.711302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:41.711302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 03:26:41.711302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:41.712140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 79/151] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:41 2022] GET /identity/v3/roles/reader => generated 84 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:26:41.719356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646e4b99-05c3-4aba-9824-c7b08c7e9559 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:41.785586 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 73/152] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:26:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 505 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:26:41.786516 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2c1e35-bd07-4972-9512-f96099c37d04 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:41.873620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:41.909171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2c1e35-bd07-4972-9512-f96099c37d04 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:41.939505 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646e4b99-05c3-4aba-9824-c7b08c7e9559 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:41.966527 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85dac16-a298-49dd-8d0e-0d4ea9a1cefd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:41.967196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85dac16-a298-49dd-8d0e-0d4ea9a1cefd None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:41.967478 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85dac16-a298-49dd-8d0e-0d4ea9a1cefd None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:41.967751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85dac16-a298-49dd-8d0e-0d4ea9a1cefd None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:41.968420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85dac16-a298-49dd-8d0e-0d4ea9a1cefd None admin] RBAC: Adding query filter params (name=reader) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:41.973118 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85dac16-a298-49dd-8d0e-0d4ea9a1cefd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:41.973118 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85dac16-a298-49dd-8d0e-0d4ea9a1cefd None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:41.984453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 74/153] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:26:41 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:41.986725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85dac16-a298-49dd-8d0e-0d4ea9a1cefd None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:41.993991 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcc5e21-d0af-42e7-9f8b-a922d2963644 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:41.994647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcc5e21-d0af-42e7-9f8b-a922d2963644 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:41.994988 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcc5e21-d0af-42e7-9f8b-a922d2963644 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:41.995220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:41.995326 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcc5e21-d0af-42e7-9f8b-a922d2963644 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:42.014332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfcc5e21-d0af-42e7-9f8b-a922d2963644 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:42.020767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-dfcc5e21-d0af-42e7-9f8b-a922d2963644 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:26:42.020767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:42.020767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:42.020767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:42.020767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:42.020767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:42.020767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:42.020767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:42.020767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:42.020767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:42.020767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:42.020767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:42.020767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:42.020767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:42.020767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:42.020767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:42.021965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:42.021965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:42.021965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:42.021965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:42.021965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:42.021965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:42.021965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:42.021965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:42.021965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:42.021965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:42.021965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:42.021965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:42.021965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:42.021965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:42.021965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:42.021965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:42.022749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:42.022749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:42.022749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:42.022749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:42.022749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:42.022749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:42.022749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:42.022749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:42.022749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:42.022749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:42.022749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:26:42.022749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:42.024959 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 80/154] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:26:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:42.026530 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dda5c6-99c7-4f40-bbc5-03c5120c843b None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:42.027080 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dda5c6-99c7-4f40-bbc5-03c5120c843b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:42.027635 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dda5c6-99c7-4f40-bbc5-03c5120c843b None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:42.033369 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 81/155] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:42.034590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dda5c6-99c7-4f40-bbc5-03c5120c843b None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:42.171565 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dda5c6-99c7-4f40-bbc5-03c5120c843b None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:42.202232 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:42.212754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030a4a69-1f99-4006-94f3-9c0ff8199c78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:42.213663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030a4a69-1f99-4006-94f3-9c0ff8199c78 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:42.214114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030a4a69-1f99-4006-94f3-9c0ff8199c78 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:42.215733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030a4a69-1f99-4006-94f3-9c0ff8199c78 None admin] PATH_INFO: `/v3/users/af186a335cba418dbdfe79d5fb7a055c` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:42.216590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030a4a69-1f99-4006-94f3-9c0ff8199c78 None admin] RBAC: Authorizing `identity:get_user(user_id=af186a335cba418dbdfe79d5fb7a055c)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:42.218426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030a4a69-1f99-4006-94f3-9c0ff8199c78 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:42.220337 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 82/156] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:42 2022] GET /identity/v3/users/af186a335cba418dbdfe79d5fb7a055c => generated 289 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:42.221600 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030a4a69-1f99-4006-94f3-9c0ff8199c78 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:42.226826 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85dac16-a298-49dd-8d0e-0d4ea9a1cefd None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:42.312168 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0360f8-5aff-4278-b5ce-30a792ea18b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:42.313152 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0360f8-5aff-4278-b5ce-30a792ea18b3 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:42.313631 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0360f8-5aff-4278-b5ce-30a792ea18b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:42.314170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0360f8-5aff-4278-b5ce-30a792ea18b3 None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:42.315630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0360f8-5aff-4278-b5ce-30a792ea18b3 None admin] RBAC: Authorizing `identity:get_project(project_id=8d00e05132b54e9b856788c06cdb7a56)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:42.317386 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0360f8-5aff-4278-b5ce-30a792ea18b3 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:42.319825 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 75/157] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:41 2022] GET /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56 => generated 298 bytes in 335 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:42.325415 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23deebeb-3a9f-4179-9e9f-4829a89b9980 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:42.328566 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23deebeb-3a9f-4179-9e9f-4829a89b9980 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:42.329140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23deebeb-3a9f-4179-9e9f-4829a89b9980 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:42.354668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-23deebeb-3a9f-4179-9e9f-4829a89b9980 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:42.358339 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-23deebeb-3a9f-4179-9e9f-4829a89b9980 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:42.371275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030a4a69-1f99-4006-94f3-9c0ff8199c78 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:42.415576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483cf547-2dd8-4728-bd8d-6ddd262f73ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:42.416198 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483cf547-2dd8-4728-bd8d-6ddd262f73ad None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:42.416477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483cf547-2dd8-4728-bd8d-6ddd262f73ad None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:42.416745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483cf547-2dd8-4728-bd8d-6ddd262f73ad None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:42.417406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483cf547-2dd8-4728-bd8d-6ddd262f73ad None admin] RBAC: Adding query filter params (name=admin) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:42.417709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483cf547-2dd8-4728-bd8d-6ddd262f73ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:42.419942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483cf547-2dd8-4728-bd8d-6ddd262f73ad None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:42.443730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 83/158] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:42.445218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e7165a-0888-4705-a107-eb28d1be7db0 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:42.445499 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e7165a-0888-4705-a107-eb28d1be7db0 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:42.445771 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e7165a-0888-4705-a107-eb28d1be7db0 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:42.447143 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 84/159] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:42.449888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e7165a-0888-4705-a107-eb28d1be7db0 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:42.492856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:42.501846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 76/160] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:42.502699 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23deebeb-3a9f-4179-9e9f-4829a89b9980 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:42.578974 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:42.667262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e7165a-0888-4705-a107-eb28d1be7db0 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:42.673866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23deebeb-3a9f-4179-9e9f-4829a89b9980 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:42.736982 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0d6a44-48e9-4912-9604-fb8cafd1f22d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:42.737932 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0d6a44-48e9-4912-9604-fb8cafd1f22d None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:42.738266 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0d6a44-48e9-4912-9604-fb8cafd1f22d None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:42.738557 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0d6a44-48e9-4912-9604-fb8cafd1f22d None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:42.739771 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0d6a44-48e9-4912-9604-fb8cafd1f22d None admin] RBAC: Authorizing `identity:get_project(project_id=8d00e05132b54e9b856788c06cdb7a56)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:42.742194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0d6a44-48e9-4912-9604-fb8cafd1f22d None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:42.743886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 85/161] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:42 2022] GET /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56 => generated 298 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:42.744659 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0d6a44-48e9-4912-9604-fb8cafd1f22d None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:42.751999 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b4e21e-9653-4b50-a580-95925dae4365 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:42.752653 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b4e21e-9653-4b50-a580-95925dae4365 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:42.752932 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b4e21e-9653-4b50-a580-95925dae4365 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:42.753206 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b4e21e-9653-4b50-a580-95925dae4365 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:42.754168 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b4e21e-9653-4b50-a580-95925dae4365 None admin] RBAC: Adding query filter params (role.id=589e1f1333a042b7a00c53384a2f5426, scope.project.id=8d00e05132b54e9b856788c06cdb7a56, user.id=5eef2e8ac0a34e458d28dc3d163cb041) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:42.754480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b4e21e-9653-4b50-a580-95925dae4365 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:42.756034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b4e21e-9653-4b50-a580-95925dae4365 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:42.774297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 77/162] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:26:42 2022] GET /identity/v3/role_assignments?user.id=5eef2e8ac0a34e458d28dc3d163cb041&scope.project.id=8d00e05132b54e9b856788c06cdb7a56&role.id=589e1f1333a042b7a00c53384a2f5426 => generated 608 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:42.779879 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5142ee68-2495-4e39-9aec-428b07e967df None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:42.787434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5142ee68-2495-4e39-9aec-428b07e967df None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:42.789744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5142ee68-2495-4e39-9aec-428b07e967df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:42.836114 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-5142ee68-2495-4e39-9aec-428b07e967df None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:42.837474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5142ee68-2495-4e39-9aec-428b07e967df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:42.885277 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0d6a44-48e9-4912-9604-fb8cafd1f22d None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:42.931307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:42.942124 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3c2f48-7c37-44dd-956b-b1c53193dad7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:42.942792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3c2f48-7c37-44dd-956b-b1c53193dad7 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:42.943119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3c2f48-7c37-44dd-956b-b1c53193dad7 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:42.943394 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3c2f48-7c37-44dd-956b-b1c53193dad7 None admin] PATH_INFO: `/v3/users/3e71cde4d415471495fcc3441fcdfc54` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:42.947625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3c2f48-7c37-44dd-956b-b1c53193dad7 None admin] RBAC: Authorizing `identity:get_user(user_id=3e71cde4d415471495fcc3441fcdfc54)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:42.949171 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3c2f48-7c37-44dd-956b-b1c53193dad7 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:42.951172 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 86/163] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:42 2022] GET /identity/v3/users/3e71cde4d415471495fcc3441fcdfc54 => generated 247 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:42.951967 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3c2f48-7c37-44dd-956b-b1c53193dad7 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:42.995965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:43.013193 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 78/164] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:43.014289 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5142ee68-2495-4e39-9aec-428b07e967df None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:43.098934 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3c2f48-7c37-44dd-956b-b1c53193dad7 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:43.172327 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5142ee68-2495-4e39-9aec-428b07e967df None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:43.175281 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7779a61-93c5-4ff8-af5c-284a44f64915 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:43.176236 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7779a61-93c5-4ff8-af5c-284a44f64915 None admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:43.176716 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7779a61-93c5-4ff8-af5c-284a44f64915 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:43.177183 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7779a61-93c5-4ff8-af5c-284a44f64915 None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56/users/af186a335cba418dbdfe79d5fb7a055c/roles/363c2654ab4e456b8b9b0b8c8c28911e` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:43.182850 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7779a61-93c5-4ff8-af5c-284a44f64915 None admin] RBAC: Authorizing `identity:create_grant(project_id=8d00e05132b54e9b856788c06cdb7a56, user_id=af186a335cba418dbdfe79d5fb7a055c, role_id=363c2654ab4e456b8b9b0b8c8c28911e)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:43.184732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7779a61-93c5-4ff8-af5c-284a44f64915 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:43.212838 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908f105e-9eab-4edb-863d-7da9e515c548 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:43.215918 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908f105e-9eab-4edb-863d-7da9e515c548 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:43.215918 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908f105e-9eab-4edb-863d-7da9e515c548 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:43.215918 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908f105e-9eab-4edb-863d-7da9e515c548 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:43.281471 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 87/165] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:26:42 2022] PUT /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56/users/af186a335cba418dbdfe79d5fb7a055c/roles/363c2654ab4e456b8b9b0b8c8c28911e => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:26:43.284931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7779a61-93c5-4ff8-af5c-284a44f64915 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:43.287940 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:43.296535 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908f105e-9eab-4edb-863d-7da9e515c548 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:43.298064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908f105e-9eab-4edb-863d-7da9e515c548 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:43.401896 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:43.415319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 79/166] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:26:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:26:43.421927 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908f105e-9eab-4edb-863d-7da9e515c548 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:43.498938 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7779a61-93c5-4ff8-af5c-284a44f64915 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:43.554198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:43.561390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908f105e-9eab-4edb-863d-7da9e515c548 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:43.574267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bba8a3a-1089-4283-bf29-8fe2dfe352fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:43.575192 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bba8a3a-1089-4283-bf29-8fe2dfe352fb None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:43.575575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bba8a3a-1089-4283-bf29-8fe2dfe352fb None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:43.575948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bba8a3a-1089-4283-bf29-8fe2dfe352fb None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:43.599217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bba8a3a-1089-4283-bf29-8fe2dfe352fb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:43.604837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-6bba8a3a-1089-4283-bf29-8fe2dfe352fb None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:26:43.604837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:43.604837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:43.604837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:43.604837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:43.604837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:43.604837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:43.604837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:43.604837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:43.604837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:43.604837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:43.604837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:43.604837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:43.604837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:43.604837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:43.604837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:43.605842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:43.605842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:43.605842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:43.605842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:43.605842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:43.605842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:43.605842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:43.605842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:43.605842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:43.605842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:43.605842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:43.605842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:43.605842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:43.605842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:43.605842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:43.605842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:43.606710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:43.606710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:43.606710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:43.606710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:43.606710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:43.606710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:43.606710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:43.606710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:43.606710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:43.606710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:43.606710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:26:43.606710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:43.607336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e66f743-37f6-4861-8f91-ef8129ac056e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:43.607336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e66f743-37f6-4861-8f91-ef8129ac056e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:43.607336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e66f743-37f6-4861-8f91-ef8129ac056e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:43.607557 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e66f743-37f6-4861-8f91-ef8129ac056e None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:43.608872 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e66f743-37f6-4861-8f91-ef8129ac056e None admin] RBAC: Authorizing `identity:get_project(project_id=8d00e05132b54e9b856788c06cdb7a56)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:43.609336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 88/167] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:26:43 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 327 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:43.610654 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e66f743-37f6-4861-8f91-ef8129ac056e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:43.613047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 80/168] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:43 2022] GET /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56 => generated 298 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:26:43.616399 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bba8a3a-1089-4283-bf29-8fe2dfe352fb None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:43.631125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e66f743-37f6-4861-8f91-ef8129ac056e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:43.750674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bba8a3a-1089-4283-bf29-8fe2dfe352fb None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:43.777424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:43.814243 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd21a20-3313-481b-bfa5-786a863c7dea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:43.817226 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd21a20-3313-481b-bfa5-786a863c7dea None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:43.817226 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd21a20-3313-481b-bfa5-786a863c7dea None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:43.817226 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd21a20-3313-481b-bfa5-786a863c7dea None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:43.817467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd21a20-3313-481b-bfa5-786a863c7dea None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:43.822933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd21a20-3313-481b-bfa5-786a863c7dea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:43.822933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd21a20-3313-481b-bfa5-786a863c7dea None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:43.824605 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e66f743-37f6-4861-8f91-ef8129ac056e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:43.842585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 89/169] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:26:43 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:43.842585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd21a20-3313-481b-bfa5-786a863c7dea None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:43.890951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a136c8-f2d8-42e7-9830-5aa211313f0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:43.890951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a136c8-f2d8-42e7-9830-5aa211313f0f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:43.890951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a136c8-f2d8-42e7-9830-5aa211313f0f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:43.890951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a136c8-f2d8-42e7-9830-5aa211313f0f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:43.914941 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a136c8-f2d8-42e7-9830-5aa211313f0f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:43.920715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-c0a136c8-f2d8-42e7-9830-5aa211313f0f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:26:43.920715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:43.920715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:43.920715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:43.920715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:43.920715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:43.920715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:43.920715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:43.920715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:43.920715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:43.920715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:43.920715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:43.920715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:43.920715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:43.920715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:43.920715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:43.922574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:43.922574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:43.922574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:43.922574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:43.922574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:43.922574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:43.922574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:43.922574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:43.922574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:43.922574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:43.922574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:43.922574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:43.922574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:43.922574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:43.922574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:43.922574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:43.925186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:43.925186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:43.925186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:43.925186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:43.925186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:43.925186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:43.925186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:43.925186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:43.925186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:43.925186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:43.925186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:26:43.925186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:26:43.925186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 81/170] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:43 2022] GET /identity/v3/roles/member => generated 84 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:26:43.925186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a136c8-f2d8-42e7-9830-5aa211313f0f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:43.960506 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:44.033504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd21a20-3313-481b-bfa5-786a863c7dea None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:44.088365 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a136c8-f2d8-42e7-9830-5aa211313f0f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:44.099315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcd695c-7c8a-43e0-af1d-f45339e3693b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:44.100238 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcd695c-7c8a-43e0-af1d-f45339e3693b None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:44.100711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcd695c-7c8a-43e0-af1d-f45339e3693b None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:44.101171 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcd695c-7c8a-43e0-af1d-f45339e3693b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:44.102090 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcd695c-7c8a-43e0-af1d-f45339e3693b None admin] RBAC: Adding query filter params (role.id=89c4b3a971224d468d33d2dc411230a2, scope.project.id=8d00e05132b54e9b856788c06cdb7a56, user.id=3e71cde4d415471495fcc3441fcdfc54) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:44.102602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcd695c-7c8a-43e0-af1d-f45339e3693b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:44.104284 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcd695c-7c8a-43e0-af1d-f45339e3693b None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:44.113575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 90/171] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:26:43 2022] GET /identity/v3/role_assignments?user.id=3e71cde4d415471495fcc3441fcdfc54&scope.project.id=8d00e05132b54e9b856788c06cdb7a56&role.id=89c4b3a971224d468d33d2dc411230a2 => generated 608 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:44.119867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcd695c-7c8a-43e0-af1d-f45339e3693b None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:44.192590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7663f82b-a6ee-4820-a48a-be85a3c62098 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:44.193710 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7663f82b-a6ee-4820-a48a-be85a3c62098 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:44.194183 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7663f82b-a6ee-4820-a48a-be85a3c62098 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:44.194645 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7663f82b-a6ee-4820-a48a-be85a3c62098 None admin] PATH_INFO: `/v3/users/5eef2e8ac0a34e458d28dc3d163cb041` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:44.199723 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7663f82b-a6ee-4820-a48a-be85a3c62098 None admin] RBAC: Authorizing `identity:get_user(user_id=5eef2e8ac0a34e458d28dc3d163cb041)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:44.201492 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7663f82b-a6ee-4820-a48a-be85a3c62098 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:44.203761 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 82/172] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:43 2022] GET /identity/v3/users/5eef2e8ac0a34e458d28dc3d163cb041 => generated 275 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:44.210220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7663f82b-a6ee-4820-a48a-be85a3c62098 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:44.237815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:44.421895 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcd695c-7c8a-43e0-af1d-f45339e3693b None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:44.459263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:44.472347 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7663f82b-a6ee-4820-a48a-be85a3c62098 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:44.485579 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0231afc8-ea59-476c-9d0e-ed4794069920 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:44.486980 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0231afc8-ea59-476c-9d0e-ed4794069920 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:44.487421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0231afc8-ea59-476c-9d0e-ed4794069920 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:44.487835 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0231afc8-ea59-476c-9d0e-ed4794069920 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:44.488656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0231afc8-ea59-476c-9d0e-ed4794069920 None admin] RBAC: Adding query filter params (name=member) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:44.489098 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0231afc8-ea59-476c-9d0e-ed4794069920 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:44.491964 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0231afc8-ea59-476c-9d0e-ed4794069920 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:44.506300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 91/173] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:26:44 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 387 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:44.510684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12756e9e-e315-4c31-8717-8000bf754b11 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:44.512170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12756e9e-e315-4c31-8717-8000bf754b11 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:44.512600 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12756e9e-e315-4c31-8717-8000bf754b11 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:44.514086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 92/174] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:44.516908 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12756e9e-e315-4c31-8717-8000bf754b11 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:44.541279 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3ab691-1895-4a1c-9216-ad2abaab5b89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:44.542157 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3ab691-1895-4a1c-9216-ad2abaab5b89 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:44.542581 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3ab691-1895-4a1c-9216-ad2abaab5b89 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:44.543013 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3ab691-1895-4a1c-9216-ad2abaab5b89 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:44.638274 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3ab691-1895-4a1c-9216-ad2abaab5b89 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:44.640346 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3ab691-1895-4a1c-9216-ad2abaab5b89 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:44.730341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12756e9e-e315-4c31-8717-8000bf754b11 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:44.732421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:44.762331 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:44.774289 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 83/175] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:26:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 565 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:26:44.776210 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c949bf5-cae9-4090-a6bf-bff1fb176913 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:44.776664 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c949bf5-cae9-4090-a6bf-bff1fb176913 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:44.777071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c949bf5-cae9-4090-a6bf-bff1fb176913 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:44.807421 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-3c949bf5-cae9-4090-a6bf-bff1fb176913 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:44.808852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3c949bf5-cae9-4090-a6bf-bff1fb176913 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:44.824489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c42678-1a6e-475f-97ba-bb0893a7f70c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:44.824489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c42678-1a6e-475f-97ba-bb0893a7f70c None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:44.826720 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c42678-1a6e-475f-97ba-bb0893a7f70c None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:44.826720 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c42678-1a6e-475f-97ba-bb0893a7f70c None admin] PATH_INFO: `/v3/users/5eef2e8ac0a34e458d28dc3d163cb041` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:44.831649 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c42678-1a6e-475f-97ba-bb0893a7f70c None admin] RBAC: Authorizing `identity:get_user(user_id=5eef2e8ac0a34e458d28dc3d163cb041)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:44.833129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c42678-1a6e-475f-97ba-bb0893a7f70c None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:44.835034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 93/176] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:44 2022] GET /identity/v3/users/5eef2e8ac0a34e458d28dc3d163cb041 => generated 275 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:44.838505 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c42678-1a6e-475f-97ba-bb0893a7f70c None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:44.900956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 84/177] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:44.903256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f45e29-fb4f-4e85-a8f8-df851b6b9fc1 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:44.904124 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f45e29-fb4f-4e85-a8f8-df851b6b9fc1 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:44.904671 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f45e29-fb4f-4e85-a8f8-df851b6b9fc1 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:44.906291 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 85/178] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:44.911626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f45e29-fb4f-4e85-a8f8-df851b6b9fc1 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:45.021129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c42678-1a6e-475f-97ba-bb0893a7f70c None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:45.087341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b9f06e-4f67-4f6e-9603-f0fc1f68cdc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:45.088582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b9f06e-4f67-4f6e-9603-f0fc1f68cdc7 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:45.089758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b9f06e-4f67-4f6e-9603-f0fc1f68cdc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:45.090414 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b9f06e-4f67-4f6e-9603-f0fc1f68cdc7 None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:45.093587 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b9f06e-4f67-4f6e-9603-f0fc1f68cdc7 None admin] RBAC: Authorizing `identity:get_project(project_id=8d00e05132b54e9b856788c06cdb7a56)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:45.096185 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b9f06e-4f67-4f6e-9603-f0fc1f68cdc7 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:45.097208 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f45e29-fb4f-4e85-a8f8-df851b6b9fc1 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:45.101404 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 94/179] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:44 2022] GET /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56 => generated 298 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:45.103996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a287646-27dd-41b2-ad6e-1ae9bec53f32 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:45.107836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a287646-27dd-41b2-ad6e-1ae9bec53f32 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:45.108122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a287646-27dd-41b2-ad6e-1ae9bec53f32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:45.136100 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:45.144561 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-3a287646-27dd-41b2-ad6e-1ae9bec53f32 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:45.152892 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3a287646-27dd-41b2-ad6e-1ae9bec53f32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:45.168567 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ce4a1b-5ee8-48a2-a634-10419b2be3c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:45.169192 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ce4a1b-5ee8-48a2-a634-10419b2be3c7 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:45.169472 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ce4a1b-5ee8-48a2-a634-10419b2be3c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:45.169746 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ce4a1b-5ee8-48a2-a634-10419b2be3c7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:45.179377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:45.239398 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 95/180] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:45.240835 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a287646-27dd-41b2-ad6e-1ae9bec53f32 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:45.275899 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ce4a1b-5ee8-48a2-a634-10419b2be3c7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:45.278694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ce4a1b-5ee8-48a2-a634-10419b2be3c7 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:45.353867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a287646-27dd-41b2-ad6e-1ae9bec53f32 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:45.390008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233ee426-3156-478f-a592-3560834af10e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:45.390969 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233ee426-3156-478f-a592-3560834af10e None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:45.391427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233ee426-3156-478f-a592-3560834af10e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:45.392476 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233ee426-3156-478f-a592-3560834af10e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:45.393376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233ee426-3156-478f-a592-3560834af10e None admin] RBAC: Adding query filter params (role.id=fa8bdf568182483aac8a14ef98a12a95, scope.project.id=8d00e05132b54e9b856788c06cdb7a56, user.id=5eef2e8ac0a34e458d28dc3d163cb041) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:45.393909 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233ee426-3156-478f-a592-3560834af10e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:45.398368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233ee426-3156-478f-a592-3560834af10e None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:45.407940 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 96/181] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:26:45 2022] GET /identity/v3/role_assignments?user.id=5eef2e8ac0a34e458d28dc3d163cb041&scope.project.id=8d00e05132b54e9b856788c06cdb7a56&role.id=fa8bdf568182483aac8a14ef98a12a95 => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:45.409173 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233ee426-3156-478f-a592-3560834af10e None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:45.434841 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 86/182] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:26:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 528 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:26:45.436458 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2da3a0d-b5fa-433f-924b-caa776ff5c33 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:45.437696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2da3a0d-b5fa-433f-924b-caa776ff5c33 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:45.438172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2da3a0d-b5fa-433f-924b-caa776ff5c33 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:45.440935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 87/183] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:45 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:45.443326 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2da3a0d-b5fa-433f-924b-caa776ff5c33 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:45.482523 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:45.484977 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:45.604271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233ee426-3156-478f-a592-3560834af10e None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:45.606629 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2da3a0d-b5fa-433f-924b-caa776ff5c33 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:45.648094 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa66f65-b8b7-4902-b8eb-5fd6e5567acc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:45.648735 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa66f65-b8b7-4902-b8eb-5fd6e5567acc None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:45.649819 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa66f65-b8b7-4902-b8eb-5fd6e5567acc None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:45.650101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa66f65-b8b7-4902-b8eb-5fd6e5567acc None admin] PATH_INFO: `/v3/projects/6e564d1b6f644f75b62ab2a98008997a` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:45.651285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa66f65-b8b7-4902-b8eb-5fd6e5567acc None admin] RBAC: Authorizing `identity:get_project(project_id=6e564d1b6f644f75b62ab2a98008997a)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:45.652734 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa66f65-b8b7-4902-b8eb-5fd6e5567acc None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:45.652899 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2ae7e5-310a-4045-8871-73bd014af01c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:45.653674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2ae7e5-310a-4045-8871-73bd014af01c None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:45.654366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2ae7e5-310a-4045-8871-73bd014af01c None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:45.654779 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 88/184] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:45 2022] GET /identity/v3/projects/6e564d1b6f644f75b62ab2a98008997a => generated 312 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:45.655190 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2ae7e5-310a-4045-8871-73bd014af01c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:45.660350 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddb7330-7bf4-4b75-9f2e-d4ab5241110d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:45.660350 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddb7330-7bf4-4b75-9f2e-d4ab5241110d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:45.660350 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddb7330-7bf4-4b75-9f2e-d4ab5241110d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:45.672622 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2ae7e5-310a-4045-8871-73bd014af01c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:45.678832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-1d2ae7e5-310a-4045-8871-73bd014af01c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:26:45.678832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:45.678832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:45.678832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:45.678832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:45.678832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:45.678832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:45.678832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:45.678832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:45.678832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:45.678832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:45.678832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:45.678832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:45.678832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:45.678832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:45.678832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:45.679766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:45.679766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:45.679766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:45.679766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:45.679766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:45.679766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:45.679766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:45.679766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:45.679766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:45.679766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:45.679766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:45.679766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:45.679766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:45.679766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:45.679766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:45.679766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:45.680566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:45.680566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:45.680566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:45.680566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:45.680566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:45.680566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:45.680566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:45.680566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:45.680566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:45.680566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:45.680566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:26:45.680566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:45.681165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-fddb7330-7bf4-4b75-9f2e-d4ab5241110d None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:45.681165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fddb7330-7bf4-4b75-9f2e-d4ab5241110d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:45.683367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 97/185] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:26:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:45.684321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2ae7e5-310a-4045-8871-73bd014af01c None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:45.757525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 89/186] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:45.763919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddb7330-7bf4-4b75-9f2e-d4ab5241110d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:45.787836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:45.794204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:45.802628 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2ae7e5-310a-4045-8871-73bd014af01c None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:45.838486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43eb7de-50f8-4719-88b6-e81eff6ad608 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:45.839148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43eb7de-50f8-4719-88b6-e81eff6ad608 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:45.839482 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43eb7de-50f8-4719-88b6-e81eff6ad608 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:45.839754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43eb7de-50f8-4719-88b6-e81eff6ad608 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:45.851751 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43eb7de-50f8-4719-88b6-e81eff6ad608 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:45.857606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-a43eb7de-50f8-4719-88b6-e81eff6ad608 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:26:45.857606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:45.857606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:45.857606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:45.857606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:45.857606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:45.857606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:45.857606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:45.857606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:45.857606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:45.857606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:45.857606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:45.857606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:45.857606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:45.857606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:45.857606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:45.858486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:45.858486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:45.858486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:45.858486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:45.858486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:45.858486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:45.858486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:45.858486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:45.858486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:45.858486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:45.858486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:45.858486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:45.858486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:45.858486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:45.858486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:45.858486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:45.859407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:45.859407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:45.859407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:45.859407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:45.859407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:45.859407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:45.859407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:45.859407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:45.859407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:45.859407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:45.859407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:26:45.859407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:45.859407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 98/187] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:26:45 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:45.860088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43eb7de-50f8-4719-88b6-e81eff6ad608 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:45.896451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddb7330-7bf4-4b75-9f2e-d4ab5241110d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:45.946799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dded0df-e9b6-46ac-a8dd-361f16bf6046 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:45.947953 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dded0df-e9b6-46ac-a8dd-361f16bf6046 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:45.948377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dded0df-e9b6-46ac-a8dd-361f16bf6046 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:45.949047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dded0df-e9b6-46ac-a8dd-361f16bf6046 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:45.950484 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dded0df-e9b6-46ac-a8dd-361f16bf6046 None admin] RBAC: Adding query filter params (role.id=589e1f1333a042b7a00c53384a2f5426, scope.project.id=6e564d1b6f644f75b62ab2a98008997a, user.id=5eef2e8ac0a34e458d28dc3d163cb041) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:45.951240 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dded0df-e9b6-46ac-a8dd-361f16bf6046 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:45.953471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dded0df-e9b6-46ac-a8dd-361f16bf6046 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:45.962779 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 90/188] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:26:45 2022] GET /identity/v3/role_assignments?user.id=5eef2e8ac0a34e458d28dc3d163cb041&scope.project.id=6e564d1b6f644f75b62ab2a98008997a&role.id=589e1f1333a042b7a00c53384a2f5426 => generated 608 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:45.971256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dded0df-e9b6-46ac-a8dd-361f16bf6046 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:45.981606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43eb7de-50f8-4719-88b6-e81eff6ad608 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:46.048992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a72bb09-38c9-433c-b4a5-eb82f2c3dfad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:46.049631 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a72bb09-38c9-433c-b4a5-eb82f2c3dfad None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:46.049914 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a72bb09-38c9-433c-b4a5-eb82f2c3dfad None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:46.050188 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a72bb09-38c9-433c-b4a5-eb82f2c3dfad None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:46.051003 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a72bb09-38c9-433c-b4a5-eb82f2c3dfad None admin] RBAC: Adding query filter params (name=admin) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:46.051368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a72bb09-38c9-433c-b4a5-eb82f2c3dfad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:46.052844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a72bb09-38c9-433c-b4a5-eb82f2c3dfad None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:46.064954 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 99/189] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:46.064954 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a72bb09-38c9-433c-b4a5-eb82f2c3dfad None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:46.085832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:46.104855 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:46.124196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dded0df-e9b6-46ac-a8dd-361f16bf6046 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:46.168422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3beb373a-edf3-4c09-b212-5291186de759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:46.169050 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3beb373a-edf3-4c09-b212-5291186de759 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:46.169332 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3beb373a-edf3-4c09-b212-5291186de759 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:46.169604 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3beb373a-edf3-4c09-b212-5291186de759 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:46.170276 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3beb373a-edf3-4c09-b212-5291186de759 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:46.170581 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3beb373a-edf3-4c09-b212-5291186de759 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:46.172018 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3beb373a-edf3-4c09-b212-5291186de759 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:46.184124 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 91/190] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:26:45 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:46.184678 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3beb373a-edf3-4c09-b212-5291186de759 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:46.191337 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a72bb09-38c9-433c-b4a5-eb82f2c3dfad None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:46.231606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9488d6d2-d532-4b51-b29b-25aea110c928 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:46.232245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9488d6d2-d532-4b51-b29b-25aea110c928 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:46.232523 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9488d6d2-d532-4b51-b29b-25aea110c928 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:46.232795 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9488d6d2-d532-4b51-b29b-25aea110c928 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:46.233903 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9488d6d2-d532-4b51-b29b-25aea110c928 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:46.235425 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9488d6d2-d532-4b51-b29b-25aea110c928 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:46.237975 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 100/191] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:26:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:46.238975 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4954a8-3621-464d-b60e-545adf76958f None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:46.239347 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4954a8-3621-464d-b60e-545adf76958f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:46.239626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4954a8-3621-464d-b60e-545adf76958f None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:46.240955 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 101/192] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:46 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:46.241509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4954a8-3621-464d-b60e-545adf76958f None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:46.304337 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3beb373a-edf3-4c09-b212-5291186de759 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:46.332718 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:46.336867 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab64fd11-665e-4ded-b26e-de0d23cd27fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:46.337506 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab64fd11-665e-4ded-b26e-de0d23cd27fb None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:46.337786 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab64fd11-665e-4ded-b26e-de0d23cd27fb None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:46.338061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab64fd11-665e-4ded-b26e-de0d23cd27fb None admin] PATH_INFO: `/v3/users/cbc6d9ed5106450b9c621bf5be7468d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:46.351276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4954a8-3621-464d-b60e-545adf76958f None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:46.351789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab64fd11-665e-4ded-b26e-de0d23cd27fb None admin] RBAC: Authorizing `identity:get_user(user_id=cbc6d9ed5106450b9c621bf5be7468d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:46.353275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab64fd11-665e-4ded-b26e-de0d23cd27fb None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:46.354920 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 92/193] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:46 2022] GET /identity/v3/users/cbc6d9ed5106450b9c621bf5be7468d5 => generated 283 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:46.355668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab64fd11-665e-4ded-b26e-de0d23cd27fb None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:46.358191 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:46.388721 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a62f840-b4d9-4ba7-8bae-13b6d9020cb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:46.389282 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a62f840-b4d9-4ba7-8bae-13b6d9020cb4 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:46.389556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a62f840-b4d9-4ba7-8bae-13b6d9020cb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:46.389830 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a62f840-b4d9-4ba7-8bae-13b6d9020cb4 None admin] PATH_INFO: `/v3/users/cbc6d9ed5106450b9c621bf5be7468d5` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:46.390948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a62f840-b4d9-4ba7-8bae-13b6d9020cb4 None admin] RBAC: Authorizing `identity:get_user(user_id=cbc6d9ed5106450b9c621bf5be7468d5)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:46.392463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a62f840-b4d9-4ba7-8bae-13b6d9020cb4 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:46.394007 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 102/194] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:46 2022] GET /identity/v3/users/cbc6d9ed5106450b9c621bf5be7468d5 => generated 283 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:46.395116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fd6715-79d5-4c92-bb41-ab13b657698f None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:46.395398 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fd6715-79d5-4c92-bb41-ab13b657698f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:46.395679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fd6715-79d5-4c92-bb41-ab13b657698f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:46.413667 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-30fd6715-79d5-4c92-bb41-ab13b657698f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:46.414831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-30fd6715-79d5-4c92-bb41-ab13b657698f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:46.459559 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab64fd11-665e-4ded-b26e-de0d23cd27fb None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:46.480411 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 103/195] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:46.483770 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fd6715-79d5-4c92-bb41-ab13b657698f None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:46.494057 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7850c167-2b71-45c7-b220-2d5885ddff30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:46.494667 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7850c167-2b71-45c7-b220-2d5885ddff30 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:46.494973 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7850c167-2b71-45c7-b220-2d5885ddff30 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:46.495248 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7850c167-2b71-45c7-b220-2d5885ddff30 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:46.496443 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7850c167-2b71-45c7-b220-2d5885ddff30 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:46.498263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7850c167-2b71-45c7-b220-2d5885ddff30 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:46.568997 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:46.587341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fd6715-79d5-4c92-bb41-ab13b657698f None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:46.591862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 93/196] 10.222.0.49 () {60 vars in 1132 bytes} [Fri Sep 16 03:26:46 2022] POST /identity/v3/users => generated 297 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:26:46.602690 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7850c167-2b71-45c7-b220-2d5885ddff30 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:46.635299 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b110a0a-ba72-4206-bcae-53feb31c3b12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:46.636441 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b110a0a-ba72-4206-bcae-53feb31c3b12 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:46.636830 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b110a0a-ba72-4206-bcae-53feb31c3b12 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:46.637518 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b110a0a-ba72-4206-bcae-53feb31c3b12 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:46.695917 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:46.730053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b110a0a-ba72-4206-bcae-53feb31c3b12 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:46.732012 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b110a0a-ba72-4206-bcae-53feb31c3b12 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:46.751616 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7850c167-2b71-45c7-b220-2d5885ddff30 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:46.796185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1a6da8-927e-421b-abc8-fd1bdda723e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:46.796806 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1a6da8-927e-421b-abc8-fd1bdda723e1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:46.797087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1a6da8-927e-421b-abc8-fd1bdda723e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:46.797364 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1a6da8-927e-421b-abc8-fd1bdda723e1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:46.823274 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:46.844404 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 104/197] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:26:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:26:46.845250 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b110a0a-ba72-4206-bcae-53feb31c3b12 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:46.876805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1a6da8-927e-421b-abc8-fd1bdda723e1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:46.878279 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1a6da8-927e-421b-abc8-fd1bdda723e1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:46.943220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:46.976038 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b110a0a-ba72-4206-bcae-53feb31c3b12 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:47.000679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 94/198] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:26:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:26:47.002624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1a6da8-927e-421b-abc8-fd1bdda723e1 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:47.026277 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f402b194-83df-4bb8-92df-19d1799d53d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:47.027359 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f402b194-83df-4bb8-92df-19d1799d53d1 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:47.028411 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f402b194-83df-4bb8-92df-19d1799d53d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:47.029139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f402b194-83df-4bb8-92df-19d1799d53d1 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:47.047794 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f402b194-83df-4bb8-92df-19d1799d53d1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:47.055220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-f402b194-83df-4bb8-92df-19d1799d53d1 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 03:26:47.055220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:47.055220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:47.055220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:47.055220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:47.055220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:47.055220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:47.055220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:47.055220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:47.055220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:47.055220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:47.055220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:47.055220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:47.055220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:47.055220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:47.055220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:47.056160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:47.056160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:47.056160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:47.056160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:47.056160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:47.056160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:47.056160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:47.056160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:47.056160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:47.056160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:47.056160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:47.056160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:47.056160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:47.056160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:47.056160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:47.056160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:47.056995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:47.056995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:47.056995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:47.056995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:47.056995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:47.056995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:47.056995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:47.056995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:47.056995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:47.056995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:47.056995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 03:26:47.056995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:47.060162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 105/199] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:46 2022] GET /identity/v3/roles/reader => generated 84 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:47.061391 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f402b194-83df-4bb8-92df-19d1799d53d1 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:47.112325 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:47.124777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1a6da8-927e-421b-abc8-fd1bdda723e1 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:47.163780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc0d660-8986-4b6a-8835-49ee89af69ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:47.164416 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc0d660-8986-4b6a-8835-49ee89af69ec None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:47.164699 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc0d660-8986-4b6a-8835-49ee89af69ec None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:47.165001 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc0d660-8986-4b6a-8835-49ee89af69ec None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:47.166121 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc0d660-8986-4b6a-8835-49ee89af69ec None admin] RBAC: Authorizing `identity:get_project(project_id=5f8ae3affd36481c8ce85ec30e7dedd9)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:47.167655 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc0d660-8986-4b6a-8835-49ee89af69ec None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:47.169291 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 95/200] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:47 2022] GET /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9 => generated 302 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:47.170058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc0d660-8986-4b6a-8835-49ee89af69ec None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:47.181294 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f402b194-83df-4bb8-92df-19d1799d53d1 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:47.201983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:47.225365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19582e1-254a-4314-8fef-f0ff6ffb6217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:47.226301 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19582e1-254a-4314-8fef-f0ff6ffb6217 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:47.226727 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19582e1-254a-4314-8fef-f0ff6ffb6217 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:47.227168 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19582e1-254a-4314-8fef-f0ff6ffb6217 None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:47.228590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19582e1-254a-4314-8fef-f0ff6ffb6217 None admin] RBAC: Authorizing `identity:get_project(project_id=5f8ae3affd36481c8ce85ec30e7dedd9)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:47.230302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19582e1-254a-4314-8fef-f0ff6ffb6217 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:47.237172 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 106/201] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:47 2022] GET /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9 => generated 302 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:47.238227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19582e1-254a-4314-8fef-f0ff6ffb6217 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:47.284963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc0d660-8986-4b6a-8835-49ee89af69ec None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:47.323653 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2072b2-fa7d-4c84-9c9d-325cb91541fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:47.324271 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2072b2-fa7d-4c84-9c9d-325cb91541fe None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:47.324551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2072b2-fa7d-4c84-9c9d-325cb91541fe None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:47.324826 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2072b2-fa7d-4c84-9c9d-325cb91541fe None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:47.325496 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2072b2-fa7d-4c84-9c9d-325cb91541fe None admin] RBAC: Adding query filter params (name=reader) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:47.325963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2072b2-fa7d-4c84-9c9d-325cb91541fe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:47.327659 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2072b2-fa7d-4c84-9c9d-325cb91541fe None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:47.342441 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 96/202] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:26:47 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:47.343376 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2072b2-fa7d-4c84-9c9d-325cb91541fe None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:47.368972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19582e1-254a-4314-8fef-f0ff6ffb6217 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:47.382387 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:47.415457 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58820aab-de0a-4938-97d7-903d2a5b8ebf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:47.416524 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58820aab-de0a-4938-97d7-903d2a5b8ebf None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:47.417436 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58820aab-de0a-4938-97d7-903d2a5b8ebf None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:47.417869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58820aab-de0a-4938-97d7-903d2a5b8ebf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:47.419080 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58820aab-de0a-4938-97d7-903d2a5b8ebf None admin] RBAC: Adding query filter params (role.id=89c4b3a971224d468d33d2dc411230a2, scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9, user.id=cbc6d9ed5106450b9c621bf5be7468d5) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:47.419649 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58820aab-de0a-4938-97d7-903d2a5b8ebf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:47.422206 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58820aab-de0a-4938-97d7-903d2a5b8ebf None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:47.435479 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 107/203] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:26:47 2022] GET /identity/v3/role_assignments?user.id=cbc6d9ed5106450b9c621bf5be7468d5&scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9&role.id=89c4b3a971224d468d33d2dc411230a2 => generated 259 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:47.436806 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58820aab-de0a-4938-97d7-903d2a5b8ebf None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:47.446241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:47.460505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2072b2-fa7d-4c84-9c9d-325cb91541fe None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:47.509414 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa754428-6d6d-4c14-bfc1-c6ff53b8f88f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:47.511503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa754428-6d6d-4c14-bfc1-c6ff53b8f88f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:47.511972 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa754428-6d6d-4c14-bfc1-c6ff53b8f88f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:47.512422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa754428-6d6d-4c14-bfc1-c6ff53b8f88f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:47.513636 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa754428-6d6d-4c14-bfc1-c6ff53b8f88f None admin] RBAC: Adding query filter params (role.id=fa8bdf568182483aac8a14ef98a12a95, scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9, user.id=cbc6d9ed5106450b9c621bf5be7468d5) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:47.514124 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa754428-6d6d-4c14-bfc1-c6ff53b8f88f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:47.516464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa754428-6d6d-4c14-bfc1-c6ff53b8f88f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:47.542499 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 97/204] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:26:47 2022] GET /identity/v3/role_assignments?user.id=cbc6d9ed5106450b9c621bf5be7468d5&scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9&role.id=fa8bdf568182483aac8a14ef98a12a95 => generated 259 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:47.606452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58820aab-de0a-4938-97d7-903d2a5b8ebf None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:47.657481 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0579d0-3aad-42a4-b9fa-58fd62dee403 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:47.659568 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0579d0-3aad-42a4-b9fa-58fd62dee403 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:47.659852 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0579d0-3aad-42a4-b9fa-58fd62dee403 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:47.660129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0579d0-3aad-42a4-b9fa-58fd62dee403 None admin] PATH_INFO: `/v3/users/af186a335cba418dbdfe79d5fb7a055c` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:47.663729 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0579d0-3aad-42a4-b9fa-58fd62dee403 None admin] RBAC: Authorizing `identity:get_user(user_id=af186a335cba418dbdfe79d5fb7a055c)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:47.665237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0579d0-3aad-42a4-b9fa-58fd62dee403 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:47.667091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 108/205] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:47 2022] GET /identity/v3/users/af186a335cba418dbdfe79d5fb7a055c => generated 289 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:47.673045 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa754428-6d6d-4c14-bfc1-c6ff53b8f88f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:47.801335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa754428-6d6d-4c14-bfc1-c6ff53b8f88f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:47.840147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d9d3ae-922c-4064-8ad8-9e55b64cc32d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:47.840798 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d9d3ae-922c-4064-8ad8-9e55b64cc32d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:47.841080 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d9d3ae-922c-4064-8ad8-9e55b64cc32d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:47.841354 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d9d3ae-922c-4064-8ad8-9e55b64cc32d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:47.867289 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:47.940127 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d9d3ae-922c-4064-8ad8-9e55b64cc32d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:47.941869 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d9d3ae-922c-4064-8ad8-9e55b64cc32d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:48.104290 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 98/206] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:26:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:26:48.109686 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0579d0-3aad-42a4-b9fa-58fd62dee403 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:48.167492 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:48.249684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0579d0-3aad-42a4-b9fa-58fd62dee403 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:48.296993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9371df78-ff24-4723-baa9-82a6c6997975 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:48.297634 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9371df78-ff24-4723-baa9-82a6c6997975 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:48.297915 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9371df78-ff24-4723-baa9-82a6c6997975 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:48.298758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9371df78-ff24-4723-baa9-82a6c6997975 None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:48.302047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9371df78-ff24-4723-baa9-82a6c6997975 None admin] RBAC: Authorizing `identity:get_project(project_id=8d00e05132b54e9b856788c06cdb7a56)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:48.303606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9371df78-ff24-4723-baa9-82a6c6997975 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:48.305407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 109/207] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:48 2022] GET /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56 => generated 298 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:48.311209 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d9d3ae-922c-4064-8ad8-9e55b64cc32d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:48.382879 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:48.450552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d9d3ae-922c-4064-8ad8-9e55b64cc32d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:48.506060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39053c0e-e3e3-4473-bfc6-3317460dcb1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:48.506680 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39053c0e-e3e3-4473-bfc6-3317460dcb1b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:48.506986 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39053c0e-e3e3-4473-bfc6-3317460dcb1b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:48.507265 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39053c0e-e3e3-4473-bfc6-3317460dcb1b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:48.508002 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39053c0e-e3e3-4473-bfc6-3317460dcb1b None admin] RBAC: Adding query filter params (role.id=363c2654ab4e456b8b9b0b8c8c28911e, scope.project.id=8d00e05132b54e9b856788c06cdb7a56, user.id=af186a335cba418dbdfe79d5fb7a055c) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:48.508309 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39053c0e-e3e3-4473-bfc6-3317460dcb1b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:48.509766 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39053c0e-e3e3-4473-bfc6-3317460dcb1b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:48.520085 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 99/208] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:26:48 2022] GET /identity/v3/role_assignments?user.id=af186a335cba418dbdfe79d5fb7a055c&scope.project.id=8d00e05132b54e9b856788c06cdb7a56&role.id=363c2654ab4e456b8b9b0b8c8c28911e => generated 608 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:49.155783 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39662d4a-6f57-4aac-a206-a57862effedb None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:49.156322 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39662d4a-6f57-4aac-a206-a57862effedb None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:49.156322 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39662d4a-6f57-4aac-a206-a57862effedb None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:49.157916 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 110/209] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:49.163359 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669ce819-4376-4795-a653-ea2b989dae33 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:49.163878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669ce819-4376-4795-a653-ea2b989dae33 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:49.164374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669ce819-4376-4795-a653-ea2b989dae33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:49.183659 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-669ce819-4376-4795-a653-ea2b989dae33 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:49.185172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-669ce819-4376-4795-a653-ea2b989dae33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:49.260718 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 100/210] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:49.343900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19e6292-dfe7-4bef-931b-01be04842ec7 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:49.344217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19e6292-dfe7-4bef-931b-01be04842ec7 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:49.344495 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19e6292-dfe7-4bef-931b-01be04842ec7 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:49.346491 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 111/211] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:49.351904 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c28abb-729b-43f9-85b1-0505653589de None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:49.352188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c28abb-729b-43f9-85b1-0505653589de None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:49.352463 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c28abb-729b-43f9-85b1-0505653589de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:49.370696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-c7c28abb-729b-43f9-85b1-0505653589de None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:49.371899 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c7c28abb-729b-43f9-85b1-0505653589de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:49.416315 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:49.445235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 101/212] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:49.558429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19e6292-dfe7-4bef-931b-01be04842ec7 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:49.662116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19e6292-dfe7-4bef-931b-01be04842ec7 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:49.671933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:49.696758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99db9ed2-74a0-4eb8-b103-13b9389117df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:49.697396 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99db9ed2-74a0-4eb8-b103-13b9389117df None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:49.697676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99db9ed2-74a0-4eb8-b103-13b9389117df None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:49.697950 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99db9ed2-74a0-4eb8-b103-13b9389117df None admin] PATH_INFO: `/v3/domains/default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:49.699090 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99db9ed2-74a0-4eb8-b103-13b9389117df None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:49.700530 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99db9ed2-74a0-4eb8-b103-13b9389117df None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:49.702380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 112/213] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:26:49 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:49.708511 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c28abb-729b-43f9-85b1-0505653589de None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:49.736638 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99db9ed2-74a0-4eb8-b103-13b9389117df None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:49.822861 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c28abb-729b-43f9-85b1-0505653589de None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:49.857278 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99db9ed2-74a0-4eb8-b103-13b9389117df None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:49.862867 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60178463-f325-4827-b2ec-57331294dc4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:49.863604 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60178463-f325-4827-b2ec-57331294dc4d None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:49.863953 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60178463-f325-4827-b2ec-57331294dc4d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:49.864289 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60178463-f325-4827-b2ec-57331294dc4d None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:49.865009 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60178463-f325-4827-b2ec-57331294dc4d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:49.867020 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60178463-f325-4827-b2ec-57331294dc4d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:49.898371 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f777d2-0f67-43f1-965f-0083ffa938c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:49.899072 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f777d2-0f67-43f1-965f-0083ffa938c7 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:49.899352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f777d2-0f67-43f1-965f-0083ffa938c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:49.899622 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f777d2-0f67-43f1-965f-0083ffa938c7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:49.914011 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f777d2-0f67-43f1-965f-0083ffa938c7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:49.919304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-32f777d2-0f67-43f1-965f-0083ffa938c7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:26:49.919304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:49.919304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:49.919304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:49.919304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:49.919304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:49.919304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:49.919304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:49.919304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:49.919304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:49.919304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:49.919304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:49.919304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:49.919304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:49.919304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:49.919304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:49.920242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:49.920242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:49.920242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:49.920242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:49.920242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:49.920242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:49.920242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:49.920242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:49.920242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:49.920242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:49.920242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:49.920242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:49.920242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:49.920242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:49.920242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:49.920242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:49.921047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:49.921047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:49.921047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:49.921047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:49.921047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:49.921047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:49.921047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:49.921047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:49.921047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:49.921047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:49.921047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:26:49.921047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:49.921047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 113/214] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:49.927039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f777d2-0f67-43f1-965f-0083ffa938c7 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:49.964421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:49.978723 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 102/215] 10.222.0.49 () {60 vars in 1132 bytes} [Fri Sep 16 03:26:49 2022] POST /identity/v3/users => generated 297 bytes in 264 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:26:50.026858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4be0b6-913c-4edf-b3c5-f2a7bfd74ed1 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:50.028744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4be0b6-913c-4edf-b3c5-f2a7bfd74ed1 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:50.029505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4be0b6-913c-4edf-b3c5-f2a7bfd74ed1 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:50.031233 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 103/216] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:50 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:26:50.038314 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31361a01-fd3a-40f8-bf1d-51a29e397866 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:50.038314 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31361a01-fd3a-40f8-bf1d-51a29e397866 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:50.038314 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31361a01-fd3a-40f8-bf1d-51a29e397866 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:50.045689 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f777d2-0f67-43f1-965f-0083ffa938c7 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:50.065368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-31361a01-fd3a-40f8-bf1d-51a29e397866 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:50.066789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-31361a01-fd3a-40f8-bf1d-51a29e397866 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:50.097249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ef8176-25f6-4692-a427-8293e9ae33a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:50.098833 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ef8176-25f6-4692-a427-8293e9ae33a9 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:50.099365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ef8176-25f6-4692-a427-8293e9ae33a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:50.100040 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ef8176-25f6-4692-a427-8293e9ae33a9 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:50.101198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:50.101609 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ef8176-25f6-4692-a427-8293e9ae33a9 None admin] RBAC: Adding query filter params (name=member) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:50.102721 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ef8176-25f6-4692-a427-8293e9ae33a9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:50.104675 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ef8176-25f6-4692-a427-8293e9ae33a9 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:50.116649 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 114/217] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:26:49 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:26:50.122895 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ef8176-25f6-4692-a427-8293e9ae33a9 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:50.163022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 104/218] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 03:26:50.236919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:50.240543 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ef8176-25f6-4692-a427-8293e9ae33a9 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:50.275944 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44b08de-3f27-4299-9a92-0e86fb8f5edd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:50.276592 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44b08de-3f27-4299-9a92-0e86fb8f5edd None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:50.276878 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44b08de-3f27-4299-9a92-0e86fb8f5edd None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:50.277155 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44b08de-3f27-4299-9a92-0e86fb8f5edd None admin] PATH_INFO: `/v3/users/09371fdca3714892af909c965686fcaf` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:50.293808 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44b08de-3f27-4299-9a92-0e86fb8f5edd None admin] RBAC: Authorizing `identity:get_user(user_id=09371fdca3714892af909c965686fcaf)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:50.295501 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44b08de-3f27-4299-9a92-0e86fb8f5edd None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:50.297669 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 115/219] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:50 2022] GET /identity/v3/users/09371fdca3714892af909c965686fcaf => generated 297 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:26:50.303432 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31361a01-fd3a-40f8-bf1d-51a29e397866 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:50.426206 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31361a01-fd3a-40f8-bf1d-51a29e397866 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:50.434001 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc38950f-cb2e-49b1-8bce-b78a9aa4ac92 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:50.434280 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc38950f-cb2e-49b1-8bce-b78a9aa4ac92 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:50.434553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc38950f-cb2e-49b1-8bce-b78a9aa4ac92 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:50.436436 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 116/220] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:50.442722 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc38950f-cb2e-49b1-8bce-b78a9aa4ac92 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:50.468514 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6daed59-49b4-4ff5-bcbd-b25659b75ea2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:50.469256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6daed59-49b4-4ff5-bcbd-b25659b75ea2 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:50.469535 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6daed59-49b4-4ff5-bcbd-b25659b75ea2 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:50.469806 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6daed59-49b4-4ff5-bcbd-b25659b75ea2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:50.504112 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:50.556299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6daed59-49b4-4ff5-bcbd-b25659b75ea2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:50.558630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6daed59-49b4-4ff5-bcbd-b25659b75ea2 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:50.569958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc38950f-cb2e-49b1-8bce-b78a9aa4ac92 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:50.609863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473ac035-8c40-4bd0-8a19-8eb5bc6bf111 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:50.611466 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473ac035-8c40-4bd0-8a19-8eb5bc6bf111 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:50.611975 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473ac035-8c40-4bd0-8a19-8eb5bc6bf111 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:50.612434 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473ac035-8c40-4bd0-8a19-8eb5bc6bf111 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:50.624624 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473ac035-8c40-4bd0-8a19-8eb5bc6bf111 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:50.630308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-473ac035-8c40-4bd0-8a19-8eb5bc6bf111 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:26:50.630308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:50.630308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:50.630308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:50.630308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:50.630308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:50.630308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:50.630308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:50.630308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:50.630308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:50.630308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:50.630308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:50.630308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:50.630308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:50.630308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:50.630308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:50.631462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:50.631462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:50.631462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:50.631462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:50.631462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:50.631462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:50.631462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:50.631462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:50.631462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:50.631462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:50.631462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:50.631462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:50.631462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:50.631462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:50.631462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:50.631462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:50.632489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:50.632489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:50.632489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:50.632489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:50.632489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:50.632489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:50.632489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:50.632489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:50.632489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:50.632489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:50.632489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:26:50.632489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:50.634893 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 117/221] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:26:50 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:26:50.641292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8a9d3d-28d1-4903-8a50-d5ac276a0f9e None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:50.641818 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8a9d3d-28d1-4903-8a50-d5ac276a0f9e None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:50.642871 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8a9d3d-28d1-4903-8a50-d5ac276a0f9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:50.662778 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-1f8a9d3d-28d1-4903-8a50-d5ac276a0f9e None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:50.664599 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1f8a9d3d-28d1-4903-8a50-d5ac276a0f9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:50.678306 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:50.693207 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 105/222] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:26:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:26:50.694171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6daed59-49b4-4ff5-bcbd-b25659b75ea2 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:50.741659 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 118/223] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:50.742352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8a9d3d-28d1-4903-8a50-d5ac276a0f9e None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:50.811512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:50.884120 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6daed59-49b4-4ff5-bcbd-b25659b75ea2 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:50.926644 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8a9d3d-28d1-4903-8a50-d5ac276a0f9e None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:50.927613 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6d20c4-8698-434b-8641-5816af9f3dec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:50.928487 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6d20c4-8698-434b-8641-5816af9f3dec None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:50.928968 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6d20c4-8698-434b-8641-5816af9f3dec None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:50.929436 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6d20c4-8698-434b-8641-5816af9f3dec None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:50.931763 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6d20c4-8698-434b-8641-5816af9f3dec None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:50.932273 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6d20c4-8698-434b-8641-5816af9f3dec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:50.934080 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6d20c4-8698-434b-8641-5816af9f3dec None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:50.948385 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 106/224] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:26:50 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:50.954059 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6d20c4-8698-434b-8641-5816af9f3dec None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:50.976468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfbdfd6-c6e1-4d38-b7b4-631760f30257 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:50.983911 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfbdfd6-c6e1-4d38-b7b4-631760f30257 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:50.983911 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfbdfd6-c6e1-4d38-b7b4-631760f30257 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:50.983911 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfbdfd6-c6e1-4d38-b7b4-631760f30257 None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:50.983911 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfbdfd6-c6e1-4d38-b7b4-631760f30257 None admin] RBAC: Authorizing `identity:get_project(project_id=5f8ae3affd36481c8ce85ec30e7dedd9)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:50.983911 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfbdfd6-c6e1-4d38-b7b4-631760f30257 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:50.983911 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 119/225] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:50 2022] GET /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9 => generated 302 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:50.988394 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfbdfd6-c6e1-4d38-b7b4-631760f30257 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:51.005609 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:51.129493 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfbdfd6-c6e1-4d38-b7b4-631760f30257 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:51.152177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6d20c4-8698-434b-8641-5816af9f3dec None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:51.177132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14cee55-9c82-445e-aa7b-ebf289897537 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:51.178360 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:51.179149 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14cee55-9c82-445e-aa7b-ebf289897537 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:51.179692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14cee55-9c82-445e-aa7b-ebf289897537 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:51.180253 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14cee55-9c82-445e-aa7b-ebf289897537 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:51.181237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14cee55-9c82-445e-aa7b-ebf289897537 None admin] RBAC: Adding query filter params (role.id=589e1f1333a042b7a00c53384a2f5426, scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9, user.id=09371fdca3714892af909c965686fcaf) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:51.181873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14cee55-9c82-445e-aa7b-ebf289897537 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:51.184180 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14cee55-9c82-445e-aa7b-ebf289897537 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:51.191563 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483f205f-cb70-4223-951e-48d469058e62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:51.192442 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483f205f-cb70-4223-951e-48d469058e62 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:51.192828 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 120/226] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:26:50 2022] GET /identity/v3/role_assignments?user.id=09371fdca3714892af909c965686fcaf&scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9&role.id=589e1f1333a042b7a00c53384a2f5426 => generated 259 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:51.193209 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483f205f-cb70-4223-951e-48d469058e62 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:51.194091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14cee55-9c82-445e-aa7b-ebf289897537 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:51.194470 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483f205f-cb70-4223-951e-48d469058e62 None admin] PATH_INFO: `/v3/users/cbc6d9ed5106450b9c621bf5be7468d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:51.199924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483f205f-cb70-4223-951e-48d469058e62 None admin] RBAC: Authorizing `identity:get_user(user_id=cbc6d9ed5106450b9c621bf5be7468d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:51.204306 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483f205f-cb70-4223-951e-48d469058e62 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:51.208051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 107/227] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:50 2022] GET /identity/v3/users/cbc6d9ed5106450b9c621bf5be7468d5 => generated 283 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:26:51.215527 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483f205f-cb70-4223-951e-48d469058e62 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:51.372551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:51.377259 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483f205f-cb70-4223-951e-48d469058e62 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:51.414756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14cee55-9c82-445e-aa7b-ebf289897537 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:51.422732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b4f3c7-523f-4669-bfc8-206883555420 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:51.424337 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b4f3c7-523f-4669-bfc8-206883555420 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:51.424829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b4f3c7-523f-4669-bfc8-206883555420 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:51.425306 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b4f3c7-523f-4669-bfc8-206883555420 None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:51.427518 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b4f3c7-523f-4669-bfc8-206883555420 None admin] RBAC: Authorizing `identity:get_project(project_id=5f8ae3affd36481c8ce85ec30e7dedd9)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:51.429320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b4f3c7-523f-4669-bfc8-206883555420 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:51.431310 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 108/228] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:51 2022] GET /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9 => generated 302 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:51.437802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b4f3c7-523f-4669-bfc8-206883555420 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:51.481396 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f1c3b3-80e0-4e00-a99f-c6e414426011 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:51.482074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f1c3b3-80e0-4e00-a99f-c6e414426011 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:51.482381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f1c3b3-80e0-4e00-a99f-c6e414426011 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:51.482637 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f1c3b3-80e0-4e00-a99f-c6e414426011 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:51.499684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f1c3b3-80e0-4e00-a99f-c6e414426011 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:51.507139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-84f1c3b3-80e0-4e00-a99f-c6e414426011 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:26:51.507139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:51.507139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:51.507139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:51.507139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:51.507139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:51.507139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:51.507139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:51.507139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:51.507139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:51.507139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:51.507139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:51.507139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:51.507139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:51.507139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:51.507139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:51.508091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:51.508091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:51.508091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:51.508091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:51.508091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:51.508091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:51.508091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:51.508091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:51.508091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:51.508091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:51.508091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:51.508091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:51.508091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:51.508091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:51.508091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:51.508091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:51.508900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:51.508900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:51.508900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:51.508900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:51.508900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:51.508900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:51.508900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:51.508900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:51.508900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:51.508900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:51.508900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:26:51.508900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:51.508900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 121/229] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:26:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:51.516160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f1c3b3-80e0-4e00-a99f-c6e414426011 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:51.523333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:51.571117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b4f3c7-523f-4669-bfc8-206883555420 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:51.608487 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4526d962-4835-4c8b-ac0b-758f4b018f27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:51.609525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4526d962-4835-4c8b-ac0b-758f4b018f27 None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:51.610014 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4526d962-4835-4c8b-ac0b-758f4b018f27 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:51.610496 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4526d962-4835-4c8b-ac0b-758f4b018f27 None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9/users/cbc6d9ed5106450b9c621bf5be7468d5/roles/fa8bdf568182483aac8a14ef98a12a95` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:51.613137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4526d962-4835-4c8b-ac0b-758f4b018f27 None admin] RBAC: Authorizing `identity:create_grant(project_id=5f8ae3affd36481c8ce85ec30e7dedd9, user_id=cbc6d9ed5106450b9c621bf5be7468d5, role_id=fa8bdf568182483aac8a14ef98a12a95)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:51.614866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4526d962-4835-4c8b-ac0b-758f4b018f27 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:51.675683 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 109/230] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:26:51 2022] PUT /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9/users/cbc6d9ed5106450b9c621bf5be7468d5/roles/fa8bdf568182483aac8a14ef98a12a95 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:26:51.716604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f1c3b3-80e0-4e00-a99f-c6e414426011 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:51.770881 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7feaab5c-96bd-467c-9091-203a41f55b89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:51.771529 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feaab5c-96bd-467c-9091-203a41f55b89 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:51.771807 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feaab5c-96bd-467c-9091-203a41f55b89 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:51.772090 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feaab5c-96bd-467c-9091-203a41f55b89 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:51.772767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7feaab5c-96bd-467c-9091-203a41f55b89 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:51.773075 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7feaab5c-96bd-467c-9091-203a41f55b89 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:51.774504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7feaab5c-96bd-467c-9091-203a41f55b89 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:51.780744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:51.789991 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 122/231] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:51.791223 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4526d962-4835-4c8b-ac0b-758f4b018f27 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:51.944422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4526d962-4835-4c8b-ac0b-758f4b018f27 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:51.972378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:51.993615 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbca0a23-c55a-4a89-9cd7-416a5528134d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:51.994582 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbca0a23-c55a-4a89-9cd7-416a5528134d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:51.995075 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbca0a23-c55a-4a89-9cd7-416a5528134d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:51.995552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbca0a23-c55a-4a89-9cd7-416a5528134d None admin] PATH_INFO: `/v3/users/cbc6d9ed5106450b9c621bf5be7468d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:51.997715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbca0a23-c55a-4a89-9cd7-416a5528134d None admin] RBAC: Authorizing `identity:get_user(user_id=cbc6d9ed5106450b9c621bf5be7468d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:51.999405 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbca0a23-c55a-4a89-9cd7-416a5528134d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:52.001658 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 110/232] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:51 2022] GET /identity/v3/users/cbc6d9ed5106450b9c621bf5be7468d5 => generated 283 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:52.006876 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7feaab5c-96bd-467c-9091-203a41f55b89 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:52.147161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7feaab5c-96bd-467c-9091-203a41f55b89 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:52.182760 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4abda5e-6112-479b-93e6-2deb53696258 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:52.183429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4abda5e-6112-479b-93e6-2deb53696258 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:52.183713 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4abda5e-6112-479b-93e6-2deb53696258 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:52.183992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4abda5e-6112-479b-93e6-2deb53696258 None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:52.185122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4abda5e-6112-479b-93e6-2deb53696258 None admin] RBAC: Authorizing `identity:get_project(project_id=5f8ae3affd36481c8ce85ec30e7dedd9)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:52.186590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4abda5e-6112-479b-93e6-2deb53696258 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:52.188257 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 123/233] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:52 2022] GET /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9 => generated 302 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:52.195433 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbca0a23-c55a-4a89-9cd7-416a5528134d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:52.324501 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbca0a23-c55a-4a89-9cd7-416a5528134d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:52.357926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c22644-69c0-4dee-8dbb-aedf3310c636 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:52.358597 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c22644-69c0-4dee-8dbb-aedf3310c636 None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:52.358876 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c22644-69c0-4dee-8dbb-aedf3310c636 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:52.359178 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c22644-69c0-4dee-8dbb-aedf3310c636 None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9/users/cbc6d9ed5106450b9c621bf5be7468d5/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:52.361800 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c22644-69c0-4dee-8dbb-aedf3310c636 None admin] RBAC: Authorizing `identity:create_grant(project_id=5f8ae3affd36481c8ce85ec30e7dedd9, user_id=cbc6d9ed5106450b9c621bf5be7468d5, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:52.363290 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c22644-69c0-4dee-8dbb-aedf3310c636 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:52.490418 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 111/234] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:26:52 2022] PUT /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9/users/cbc6d9ed5106450b9c621bf5be7468d5/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:26:52.727806 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26379a90-1919-463a-9f57-e82bb3a3a8d4 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:52.728315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26379a90-1919-463a-9f57-e82bb3a3a8d4 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:52.728388 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26379a90-1919-463a-9f57-e82bb3a3a8d4 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:52.730005 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 124/235] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:52.736114 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787c71ae-ec87-46bb-9b0b-a2918d45cf46 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:52.736411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787c71ae-ec87-46bb-9b0b-a2918d45cf46 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:52.736678 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787c71ae-ec87-46bb-9b0b-a2918d45cf46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:52.756308 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-787c71ae-ec87-46bb-9b0b-a2918d45cf46 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:52.757450 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-787c71ae-ec87-46bb-9b0b-a2918d45cf46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:52.806751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:52.833554 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 112/236] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:53.027198 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d3a1f3-d27d-4dca-a527-dea1b73c647b None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:53.027496 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d3a1f3-d27d-4dca-a527-dea1b73c647b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:53.027680 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d3a1f3-d27d-4dca-a527-dea1b73c647b None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:53.029461 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 125/237] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:53.044324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a75c036-0b36-4a7f-881a-514e77546ceb None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:53.044892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a75c036-0b36-4a7f-881a-514e77546ceb None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:53.044892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a75c036-0b36-4a7f-881a-514e77546ceb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:53.067504 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2a75c036-0b36-4a7f-881a-514e77546ceb None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:53.068673 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2a75c036-0b36-4a7f-881a-514e77546ceb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:53.167251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 113/238] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:53.195242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d3a1f3-d27d-4dca-a527-dea1b73c647b None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:53.290820 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:53.352555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d3a1f3-d27d-4dca-a527-dea1b73c647b None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:53.393752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a6c4b0-faf5-4fc6-b49c-d97798dab166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:53.394390 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a6c4b0-faf5-4fc6-b49c-d97798dab166 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:53.394670 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a6c4b0-faf5-4fc6-b49c-d97798dab166 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:53.394965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a6c4b0-faf5-4fc6-b49c-d97798dab166 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:53.410782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a6c4b0-faf5-4fc6-b49c-d97798dab166 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:53.416105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-94a6c4b0-faf5-4fc6-b49c-d97798dab166 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 03:26:53.416105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:53.416105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:53.416105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:53.416105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:53.416105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:53.416105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:53.416105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:53.416105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:53.416105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:53.416105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:53.416105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:53.416105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:53.416105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:53.416105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:53.416105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:53.417008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:53.417008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:53.417008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:53.417008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:53.417008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:53.417008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:53.417008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:53.417008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:53.417008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:53.417008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:53.417008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:53.417008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:53.417008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:53.417008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:53.417008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:53.417008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:53.417815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:53.417815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:53.417815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:53.417815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:53.417815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:53.417815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:53.417815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:53.417815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:53.417815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:53.417815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:53.417815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 03:26:53.417815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:53.417815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 126/239] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:53 2022] GET /identity/v3/roles/reader => generated 84 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:53.423628 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a75c036-0b36-4a7f-881a-514e77546ceb None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:53.559505 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a6c4b0-faf5-4fc6-b49c-d97798dab166 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:53.563461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a75c036-0b36-4a7f-881a-514e77546ceb None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:53.584876 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:53.620608 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec378aa6-bcad-48d2-8f00-f897dad15d62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:53.621274 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec378aa6-bcad-48d2-8f00-f897dad15d62 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:53.621553 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec378aa6-bcad-48d2-8f00-f897dad15d62 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:53.621821 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec378aa6-bcad-48d2-8f00-f897dad15d62 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:53.622482 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec378aa6-bcad-48d2-8f00-f897dad15d62 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:53.622777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec378aa6-bcad-48d2-8f00-f897dad15d62 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:53.624174 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec378aa6-bcad-48d2-8f00-f897dad15d62 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:53.640174 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 114/240] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:26:53 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:53.646816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec378aa6-bcad-48d2-8f00-f897dad15d62 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:53.711607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a6c4b0-faf5-4fc6-b49c-d97798dab166 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:53.757859 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:53.773283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f659f2-74e0-4d98-841c-ca180a8f1edd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:53.773954 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f659f2-74e0-4d98-841c-ca180a8f1edd None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:53.774238 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f659f2-74e0-4d98-841c-ca180a8f1edd None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:53.775525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f659f2-74e0-4d98-841c-ca180a8f1edd None admin] PATH_INFO: `/v3/domains/default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:53.775845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f659f2-74e0-4d98-841c-ca180a8f1edd None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:53.777330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f659f2-74e0-4d98-841c-ca180a8f1edd None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:53.779216 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 127/241] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:26:53 2022] GET /identity/v3/domains/default => generated 199 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:26:53.787887 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f659f2-74e0-4d98-841c-ca180a8f1edd None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:53.820379 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec378aa6-bcad-48d2-8f00-f897dad15d62 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:53.859780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e53488-d841-4995-b15f-b84915c7931d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:53.861423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e53488-d841-4995-b15f-b84915c7931d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:53.861696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e53488-d841-4995-b15f-b84915c7931d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:53.861967 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e53488-d841-4995-b15f-b84915c7931d None admin] PATH_INFO: `/v3/users/38a0fa7debad4252bd89cf6377b57c5c` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:53.879177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e53488-d841-4995-b15f-b84915c7931d None admin] RBAC: Authorizing `identity:get_user(user_id=38a0fa7debad4252bd89cf6377b57c5c)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:53.880952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e53488-d841-4995-b15f-b84915c7931d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:53.883715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 115/242] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:53 2022] GET /identity/v3/users/38a0fa7debad4252bd89cf6377b57c5c => generated 297 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:53.885623 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1915e826-0bd8-4904-ad50-1597f02db129 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:53.886114 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1915e826-0bd8-4904-ad50-1597f02db129 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:53.886853 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1915e826-0bd8-4904-ad50-1597f02db129 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:53.888315 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 116/243] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:53.893247 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1915e826-0bd8-4904-ad50-1597f02db129 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:53.925159 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:53.942374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f659f2-74e0-4d98-841c-ca180a8f1edd None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:53.978495 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40389faf-3a68-475e-a246-4e2d3150985b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:53.979382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40389faf-3a68-475e-a246-4e2d3150985b None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:53.979747 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40389faf-3a68-475e-a246-4e2d3150985b None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:53.980088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40389faf-3a68-475e-a246-4e2d3150985b None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:53.980804 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40389faf-3a68-475e-a246-4e2d3150985b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:53.982320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40389faf-3a68-475e-a246-4e2d3150985b None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:54.020188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1915e826-0bd8-4904-ad50-1597f02db129 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:54.057793 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab17322-acbe-4f42-9833-9dc39b5183ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:54.059151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab17322-acbe-4f42-9833-9dc39b5183ee None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:54.059603 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab17322-acbe-4f42-9833-9dc39b5183ee None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:54.060047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab17322-acbe-4f42-9833-9dc39b5183ee None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:54.107168 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 128/244] 10.222.0.49 () {60 vars in 1132 bytes} [Fri Sep 16 03:26:53 2022] POST /identity/v3/users => generated 293 bytes in 310 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:26:54.109246 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4553ac-86dd-4b77-b0ac-5a98cba02889 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:54.109246 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4553ac-86dd-4b77-b0ac-5a98cba02889 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:54.109426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4553ac-86dd-4b77-b0ac-5a98cba02889 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:54.145577 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab17322-acbe-4f42-9833-9dc39b5183ee None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:54.147018 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-ca4553ac-86dd-4b77-b0ac-5a98cba02889 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:54.147499 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab17322-acbe-4f42-9833-9dc39b5183ee None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:54.148450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ca4553ac-86dd-4b77-b0ac-5a98cba02889 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:54.182850 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:54.280307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 117/245] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:26:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:26:54.288678 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab17322-acbe-4f42-9833-9dc39b5183ee None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:54.305037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:54.348927 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 129/246] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:54.446480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab17322-acbe-4f42-9833-9dc39b5183ee None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:54.463174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528df873-a545-4e94-8dc0-841054194982 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:54.463460 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528df873-a545-4e94-8dc0-841054194982 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:54.463740 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528df873-a545-4e94-8dc0-841054194982 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:54.465223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 130/247] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:26:54.470953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfed27e0-6377-40ca-a3c5-62ecb89a4733 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:54.471264 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfed27e0-6377-40ca-a3c5-62ecb89a4733 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:54.471505 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfed27e0-6377-40ca-a3c5-62ecb89a4733 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:54.486238 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:54.496625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-cfed27e0-6377-40ca-a3c5-62ecb89a4733 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:54.497813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-cfed27e0-6377-40ca-a3c5-62ecb89a4733 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:54.498344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352e11e7-97d7-4ec1-a930-241e22e3efe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:54.499028 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352e11e7-97d7-4ec1-a930-241e22e3efe6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:54.499311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352e11e7-97d7-4ec1-a930-241e22e3efe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:54.499587 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352e11e7-97d7-4ec1-a930-241e22e3efe6 None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:54.500839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-352e11e7-97d7-4ec1-a930-241e22e3efe6 None admin] RBAC: Authorizing `identity:get_project(project_id=5f8ae3affd36481c8ce85ec30e7dedd9)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:54.503204 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-352e11e7-97d7-4ec1-a930-241e22e3efe6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:54.505357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 118/248] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:54 2022] GET /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9 => generated 302 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:54.510898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352e11e7-97d7-4ec1-a930-241e22e3efe6 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:54.610314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 131/249] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 03:26:54.638523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352e11e7-97d7-4ec1-a930-241e22e3efe6 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:54.674822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401f7161-ed87-447a-9cf1-2f5c1c7e6915 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:54.675536 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401f7161-ed87-447a-9cf1-2f5c1c7e6915 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:54.676056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401f7161-ed87-447a-9cf1-2f5c1c7e6915 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:54.676330 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401f7161-ed87-447a-9cf1-2f5c1c7e6915 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:54.677062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401f7161-ed87-447a-9cf1-2f5c1c7e6915 None admin] RBAC: Adding query filter params (role.id=363c2654ab4e456b8b9b0b8c8c28911e, scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9, user.id=38a0fa7debad4252bd89cf6377b57c5c) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:54.677367 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401f7161-ed87-447a-9cf1-2f5c1c7e6915 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:54.679015 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401f7161-ed87-447a-9cf1-2f5c1c7e6915 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:54.684490 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfed27e0-6377-40ca-a3c5-62ecb89a4733 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:54.686172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 119/250] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:26:54 2022] GET /identity/v3/role_assignments?user.id=38a0fa7debad4252bd89cf6377b57c5c&scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9&role.id=363c2654ab4e456b8b9b0b8c8c28911e => generated 259 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:54.810733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfed27e0-6377-40ca-a3c5-62ecb89a4733 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:54.830086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:54.856035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9bc28c-d5a2-4d22-86eb-4ea417417988 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:54.856696 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9bc28c-d5a2-4d22-86eb-4ea417417988 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:54.856973 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9bc28c-d5a2-4d22-86eb-4ea417417988 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:54.857242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9bc28c-d5a2-4d22-86eb-4ea417417988 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:54.869932 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9bc28c-d5a2-4d22-86eb-4ea417417988 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:54.875094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-bf9bc28c-d5a2-4d22-86eb-4ea417417988 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:26:54.875094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:54.875094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:54.875094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:54.875094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:54.875094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:54.875094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:54.875094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:54.875094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:54.875094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:54.875094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:54.875094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:54.875094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:54.875094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:54.875094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:54.875094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:54.877937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:54.877937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:54.877937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:54.877937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:54.877937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:54.877937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:54.877937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:54.877937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:54.877937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:54.877937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:54.877937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:54.877937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:54.877937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:54.877937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:54.877937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:54.877937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:54.879897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:54.879897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:54.879897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:54.879897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:54.879897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:54.879897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:54.879897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:54.879897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:54.879897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:54.879897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:54.879897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:26:54.879897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:54.879897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 132/251] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:54.882434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401f7161-ed87-447a-9cf1-2f5c1c7e6915 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:54.927563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9bc28c-d5a2-4d22-86eb-4ea417417988 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:54.964177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:55.044161 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401f7161-ed87-447a-9cf1-2f5c1c7e6915 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:55.098893 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9bc28c-d5a2-4d22-86eb-4ea417417988 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:55.136819 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0b3bc9-2988-4590-97e9-03c0b990a047 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:55.137748 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0b3bc9-2988-4590-97e9-03c0b990a047 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:55.138199 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0b3bc9-2988-4590-97e9-03c0b990a047 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:55.138518 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2badc71b-0ecd-4288-895b-a3cfbf2f4167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:55.139269 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2badc71b-0ecd-4288-895b-a3cfbf2f4167 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:55.139603 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2badc71b-0ecd-4288-895b-a3cfbf2f4167 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:55.139781 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0b3bc9-2988-4590-97e9-03c0b990a047 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:55.140061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2badc71b-0ecd-4288-895b-a3cfbf2f4167 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:55.140805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2badc71b-0ecd-4288-895b-a3cfbf2f4167 None admin] RBAC: Adding query filter params (name=member) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:55.141194 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2badc71b-0ecd-4288-895b-a3cfbf2f4167 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:55.143619 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2badc71b-0ecd-4288-895b-a3cfbf2f4167 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:55.156258 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0b3bc9-2988-4590-97e9-03c0b990a047 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:55.162314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-9e0b3bc9-2988-4590-97e9-03c0b990a047 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:26:55.162314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:55.162314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:55.162314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:55.162314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:55.162314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:55.162314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:55.162314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:55.162314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:55.162314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:55.162314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:55.162314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:55.162314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:55.162314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:55.162314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:55.162314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:55.163313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:55.163313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:55.163313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:55.163313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:55.163313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:55.163313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:55.163313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:55.163313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:55.163313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:55.163313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:55.163313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:55.163313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:55.163313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:55.163313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:55.163313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:55.163313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:55.164106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:55.164106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:55.164106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:55.164106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:55.164106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:55.164106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:55.164106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:55.164106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:55.164106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:55.164106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:55.164106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:26:55.164106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:55.164721 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 120/252] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:26:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:55.164721 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e3ea79-2f02-4758-aed8-de576c7fdb84 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:55.164721 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e3ea79-2f02-4758-aed8-de576c7fdb84 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:55.164721 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e3ea79-2f02-4758-aed8-de576c7fdb84 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:55.164950 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 133/253] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:26:54 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:26:55.165747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 121/254] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:55.170861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0b3bc9-2988-4590-97e9-03c0b990a047 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:55.171025 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e3ea79-2f02-4758-aed8-de576c7fdb84 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:55.285438 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:55.291364 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e3ea79-2f02-4758-aed8-de576c7fdb84 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:55.331439 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:55.341242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e53cb7-9b46-4331-8ad7-d287964700e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:55.341892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e53cb7-9b46-4331-8ad7-d287964700e5 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:55.342226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e53cb7-9b46-4331-8ad7-d287964700e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:55.342506 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e53cb7-9b46-4331-8ad7-d287964700e5 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:55.346208 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e53cb7-9b46-4331-8ad7-d287964700e5 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:55.346208 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e53cb7-9b46-4331-8ad7-d287964700e5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:55.346208 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e53cb7-9b46-4331-8ad7-d287964700e5 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:55.362020 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 122/255] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:26:55 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:55.363303 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece6e1d2-e1c4-49d5-b059-630a77c487f2 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:55.363593 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece6e1d2-e1c4-49d5-b059-630a77c487f2 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:55.363937 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece6e1d2-e1c4-49d5-b059-630a77c487f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:55.400104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ece6e1d2-e1c4-49d5-b059-630a77c487f2 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:55.403104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ece6e1d2-e1c4-49d5-b059-630a77c487f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:55.451808 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0b3bc9-2988-4590-97e9-03c0b990a047 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:55.486719 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 123/256] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:55.487959 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece6e1d2-e1c4-49d5-b059-630a77c487f2 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:55.524657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172e2b96-2c6b-4e77-8d09-f05b8b876814 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:55.525294 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172e2b96-2c6b-4e77-8d09-f05b8b876814 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:55.525579 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172e2b96-2c6b-4e77-8d09-f05b8b876814 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:55.525864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172e2b96-2c6b-4e77-8d09-f05b8b876814 None admin] PATH_INFO: `/v3/users/09371fdca3714892af909c965686fcaf` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:55.527616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172e2b96-2c6b-4e77-8d09-f05b8b876814 None admin] RBAC: Authorizing `identity:get_user(user_id=09371fdca3714892af909c965686fcaf)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:55.529343 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172e2b96-2c6b-4e77-8d09-f05b8b876814 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:55.534784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 134/257] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:55 2022] GET /identity/v3/users/09371fdca3714892af909c965686fcaf => generated 297 bytes in 364 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:55.539862 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172e2b96-2c6b-4e77-8d09-f05b8b876814 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:55.596567 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece6e1d2-e1c4-49d5-b059-630a77c487f2 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:55.612476 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:55.635377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd7b2c8-1b23-4d9b-866b-c76353694abc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:55.636352 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd7b2c8-1b23-4d9b-866b-c76353694abc None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:55.636851 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd7b2c8-1b23-4d9b-866b-c76353694abc None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:55.637327 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd7b2c8-1b23-4d9b-866b-c76353694abc None admin] PATH_INFO: `/v3/users/cbc6d9ed5106450b9c621bf5be7468d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:55.639657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd7b2c8-1b23-4d9b-866b-c76353694abc None admin] RBAC: Authorizing `identity:get_user(user_id=cbc6d9ed5106450b9c621bf5be7468d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:55.642818 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd7b2c8-1b23-4d9b-866b-c76353694abc None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:55.644890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 124/258] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:55 2022] GET /identity/v3/users/cbc6d9ed5106450b9c621bf5be7468d5 => generated 283 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:55.651874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd7b2c8-1b23-4d9b-866b-c76353694abc None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:55.699299 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:55.701951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172e2b96-2c6b-4e77-8d09-f05b8b876814 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:55.796411 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8711cbe-6552-4d47-bdfa-6671b15d5cd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:55.797107 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8711cbe-6552-4d47-bdfa-6671b15d5cd7 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:55.797394 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8711cbe-6552-4d47-bdfa-6671b15d5cd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:55.797672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8711cbe-6552-4d47-bdfa-6671b15d5cd7 None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:55.800351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8711cbe-6552-4d47-bdfa-6671b15d5cd7 None admin] RBAC: Authorizing `identity:get_project(project_id=5f8ae3affd36481c8ce85ec30e7dedd9)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:55.803211 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8711cbe-6552-4d47-bdfa-6671b15d5cd7 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:55.804856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 135/259] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:55 2022] GET /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9 => generated 302 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:26:55.810943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8711cbe-6552-4d47-bdfa-6671b15d5cd7 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:55.895358 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd7b2c8-1b23-4d9b-866b-c76353694abc None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:55.966158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070dd55b-7c0c-4b30-8cad-b9599fa8cb14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:55.966798 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070dd55b-7c0c-4b30-8cad-b9599fa8cb14 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:55.967108 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070dd55b-7c0c-4b30-8cad-b9599fa8cb14 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:55.967383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070dd55b-7c0c-4b30-8cad-b9599fa8cb14 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:55.990929 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8711cbe-6552-4d47-bdfa-6671b15d5cd7 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:56.030134 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee5dae9-dc7d-471e-be8c-a27d70c153d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:56.030805 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee5dae9-dc7d-471e-be8c-a27d70c153d7 None admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:56.031161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee5dae9-dc7d-471e-be8c-a27d70c153d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:56.031444 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee5dae9-dc7d-471e-be8c-a27d70c153d7 None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9/users/09371fdca3714892af909c965686fcaf/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:56.034363 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee5dae9-dc7d-471e-be8c-a27d70c153d7 None admin] RBAC: Authorizing `identity:create_grant(project_id=5f8ae3affd36481c8ce85ec30e7dedd9, user_id=09371fdca3714892af909c965686fcaf, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:56.036195 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee5dae9-dc7d-471e-be8c-a27d70c153d7 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:56.059146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:56.079771 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070dd55b-7c0c-4b30-8cad-b9599fa8cb14 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:56.081347 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070dd55b-7c0c-4b30-8cad-b9599fa8cb14 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:56.138117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 136/260] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:26:55 2022] PUT /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9/users/09371fdca3714892af909c965686fcaf/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:26:56.138982 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee5dae9-dc7d-471e-be8c-a27d70c153d7 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:56.195695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 125/261] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:26:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 541 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:26:56.198504 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070dd55b-7c0c-4b30-8cad-b9599fa8cb14 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:56.205957 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:56.294155 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:56.324891 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070dd55b-7c0c-4b30-8cad-b9599fa8cb14 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:56.390553 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664749f4-5107-4a0f-965f-031765185dd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:56.392681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664749f4-5107-4a0f-965f-031765185dd7 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:56.393164 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664749f4-5107-4a0f-965f-031765185dd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:56.393628 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664749f4-5107-4a0f-965f-031765185dd7 None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:56.395164 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664749f4-5107-4a0f-965f-031765185dd7 None admin] RBAC: Authorizing `identity:get_project(project_id=5f8ae3affd36481c8ce85ec30e7dedd9)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:56.397283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664749f4-5107-4a0f-965f-031765185dd7 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:56.399582 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 126/262] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:56 2022] GET /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9 => generated 302 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:56.402749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee5dae9-dc7d-471e-be8c-a27d70c153d7 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:56.416429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664749f4-5107-4a0f-965f-031765185dd7 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:56.487577 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbf90c4-a4b2-49a1-b311-a5129a2b4914 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:56.488555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbf90c4-a4b2-49a1-b311-a5129a2b4914 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:56.489041 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbf90c4-a4b2-49a1-b311-a5129a2b4914 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:56.489374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbf90c4-a4b2-49a1-b311-a5129a2b4914 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:56.515741 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adbf90c4-a4b2-49a1-b311-a5129a2b4914 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:56.521730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-adbf90c4-a4b2-49a1-b311-a5129a2b4914 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:26:56.521730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:56.521730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:56.521730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:56.521730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:56.521730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:56.521730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:56.521730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:56.521730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:56.521730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:56.521730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:56.521730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:56.521730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:56.521730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:56.521730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:56.521730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:56.523515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:56.523515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:56.523515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:56.523515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:56.523515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:56.523515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:56.523515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:56.523515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:56.523515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:56.523515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:56.523515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:56.523515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:56.523515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:56.523515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:56.523515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:56.523515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:56.525108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:56.525108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:56.525108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:56.525108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:56.525108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:56.525108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:56.525108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:56.525108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:56.525108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:56.525108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:56.525108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:26:56.525108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:56.525108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 137/263] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:26:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 384 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:56.531875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbf90c4-a4b2-49a1-b311-a5129a2b4914 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:56.593243 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664749f4-5107-4a0f-965f-031765185dd7 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:56.673550 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a037ca7-8d9d-4cf6-89c3-4ad0348168b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:56.675553 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a037ca7-8d9d-4cf6-89c3-4ad0348168b1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:56.676026 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a037ca7-8d9d-4cf6-89c3-4ad0348168b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:56.676485 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a037ca7-8d9d-4cf6-89c3-4ad0348168b1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:56.677394 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a037ca7-8d9d-4cf6-89c3-4ad0348168b1 None admin] RBAC: Adding query filter params (role.id=fa8bdf568182483aac8a14ef98a12a95, scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9, user.id=cbc6d9ed5106450b9c621bf5be7468d5) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:56.677946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a037ca7-8d9d-4cf6-89c3-4ad0348168b1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:56.679627 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a037ca7-8d9d-4cf6-89c3-4ad0348168b1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:56.690123 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:56.692045 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 127/264] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:26:56 2022] GET /identity/v3/role_assignments?user.id=cbc6d9ed5106450b9c621bf5be7468d5&scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9&role.id=fa8bdf568182483aac8a14ef98a12a95 => generated 608 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:26:56.709585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbf90c4-a4b2-49a1-b311-a5129a2b4914 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:56.760774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bef1af8-a5b0-43c1-ba59-67aa8fcf8e8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:56.761758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bef1af8-a5b0-43c1-ba59-67aa8fcf8e8e None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:56.762225 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bef1af8-a5b0-43c1-ba59-67aa8fcf8e8e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:56.762726 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bef1af8-a5b0-43c1-ba59-67aa8fcf8e8e None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:56.763605 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bef1af8-a5b0-43c1-ba59-67aa8fcf8e8e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:56.764117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bef1af8-a5b0-43c1-ba59-67aa8fcf8e8e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:56.765710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bef1af8-a5b0-43c1-ba59-67aa8fcf8e8e None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:56.779436 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 138/265] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:56.785794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a037ca7-8d9d-4cf6-89c3-4ad0348168b1 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:56.812031 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:56.935331 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a037ca7-8d9d-4cf6-89c3-4ad0348168b1 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:56.978527 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665566da-9e05-4bfd-8acd-780f7e17dfca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:56.979495 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665566da-9e05-4bfd-8acd-780f7e17dfca None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:56.979948 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665566da-9e05-4bfd-8acd-780f7e17dfca None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:56.980401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665566da-9e05-4bfd-8acd-780f7e17dfca None admin] PATH_INFO: `/v3/users/cbc6d9ed5106450b9c621bf5be7468d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:56.985366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665566da-9e05-4bfd-8acd-780f7e17dfca None admin] RBAC: Authorizing `identity:get_user(user_id=cbc6d9ed5106450b9c621bf5be7468d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:56.987870 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665566da-9e05-4bfd-8acd-780f7e17dfca None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:56.990027 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 128/266] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:56 2022] GET /identity/v3/users/cbc6d9ed5106450b9c621bf5be7468d5 => generated 283 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:56.995607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bef1af8-a5b0-43c1-ba59-67aa8fcf8e8e None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:57.109729 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bef1af8-a5b0-43c1-ba59-67aa8fcf8e8e None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:57.160271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7eaa51a-9c8d-4f47-b6fc-b575dac8cd2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:57.160903 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7eaa51a-9c8d-4f47-b6fc-b575dac8cd2a None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:57.161182 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7eaa51a-9c8d-4f47-b6fc-b575dac8cd2a None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:57.161451 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7eaa51a-9c8d-4f47-b6fc-b575dac8cd2a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:57.192126 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:57.264986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7eaa51a-9c8d-4f47-b6fc-b575dac8cd2a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:57.266480 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7eaa51a-9c8d-4f47-b6fc-b575dac8cd2a None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:57.391751 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 139/267] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:26:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:26:57.397775 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665566da-9e05-4bfd-8acd-780f7e17dfca None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:57.504583 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:57.523826 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665566da-9e05-4bfd-8acd-780f7e17dfca None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:57.566385 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ce4430-7bdd-4777-9394-a0fee554b095 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:57.567399 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ce4430-7bdd-4777-9394-a0fee554b095 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:57.567854 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ce4430-7bdd-4777-9394-a0fee554b095 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:57.568293 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ce4430-7bdd-4777-9394-a0fee554b095 None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:57.569618 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ce4430-7bdd-4777-9394-a0fee554b095 None admin] RBAC: Authorizing `identity:get_project(project_id=5f8ae3affd36481c8ce85ec30e7dedd9)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:57.571263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ce4430-7bdd-4777-9394-a0fee554b095 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:57.573087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 129/268] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:57 2022] GET /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9 => generated 302 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:57.579258 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7eaa51a-9c8d-4f47-b6fc-b575dac8cd2a None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:57.648408 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:57.718348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7eaa51a-9c8d-4f47-b6fc-b575dac8cd2a None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:57.773318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a679eede-4caa-4ec1-b016-c120d8793dc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:57.773972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a679eede-4caa-4ec1-b016-c120d8793dc6 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:57.774248 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a679eede-4caa-4ec1-b016-c120d8793dc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:57.774526 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a679eede-4caa-4ec1-b016-c120d8793dc6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:57.775314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a679eede-4caa-4ec1-b016-c120d8793dc6 None admin] RBAC: Adding query filter params (role.id=89c4b3a971224d468d33d2dc411230a2, scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9, user.id=cbc6d9ed5106450b9c621bf5be7468d5) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:57.775627 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a679eede-4caa-4ec1-b016-c120d8793dc6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:57.777261 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a679eede-4caa-4ec1-b016-c120d8793dc6 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:57.788622 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 140/269] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:26:57 2022] GET /identity/v3/role_assignments?user.id=cbc6d9ed5106450b9c621bf5be7468d5&scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9&role.id=89c4b3a971224d468d33d2dc411230a2 => generated 608 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:57.850713 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18131a6d-4dde-4c98-89bd-825a31177b64 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:57.851345 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18131a6d-4dde-4c98-89bd-825a31177b64 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:57.851856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18131a6d-4dde-4c98-89bd-825a31177b64 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:57.853424 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 130/270] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:57.859402 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfb3de0-5482-4991-8d76-3f5ef7a50663 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:57.859692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfb3de0-5482-4991-8d76-3f5ef7a50663 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:57.859973 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfb3de0-5482-4991-8d76-3f5ef7a50663 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:57.887672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-fcfb3de0-5482-4991-8d76-3f5ef7a50663 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:57.888976 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-fcfb3de0-5482-4991-8d76-3f5ef7a50663 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:57.925126 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa602927-6bb1-40c2-95d1-dbfe95220368 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:57.925371 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa602927-6bb1-40c2-95d1-dbfe95220368 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:57.925774 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa602927-6bb1-40c2-95d1-dbfe95220368 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:57.927533 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 131/271] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:57 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:57.933205 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440a97eb-1c15-48e1-9b2d-dc13699e98bd None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:57.933555 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440a97eb-1c15-48e1-9b2d-dc13699e98bd None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:57.933902 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440a97eb-1c15-48e1-9b2d-dc13699e98bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:57.955187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-440a97eb-1c15-48e1-9b2d-dc13699e98bd None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:57.957010 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-440a97eb-1c15-48e1-9b2d-dc13699e98bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:57.984089 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 141/272] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 03:26:57.985406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29c2c51-f13f-47d5-9a69-5135894fd97e None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:57.986372 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29c2c51-f13f-47d5-9a69-5135894fd97e None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:57.986665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29c2c51-f13f-47d5-9a69-5135894fd97e None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:57.988616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 142/273] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:57.994155 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7725aa8-62cc-480c-b5c5-1aa3a32927c0 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:57.994433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7725aa8-62cc-480c-b5c5-1aa3a32927c0 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:57.994700 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7725aa8-62cc-480c-b5c5-1aa3a32927c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:58.013291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-e7725aa8-62cc-480c-b5c5-1aa3a32927c0 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:58.015201 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e7725aa8-62cc-480c-b5c5-1aa3a32927c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:58.056202 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 132/274] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 03:26:58.058162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:58.085303 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 143/275] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 03:26:58.302735 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440a97eb-1c15-48e1-9b2d-dc13699e98bd None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:58.343756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7725aa8-62cc-480c-b5c5-1aa3a32927c0 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:58.423170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:58.455942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440a97eb-1c15-48e1-9b2d-dc13699e98bd None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:58.493108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7725aa8-62cc-480c-b5c5-1aa3a32927c0 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:58.504397 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d3c597-c844-4270-8d40-7983096cdc81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:58.505061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d3c597-c844-4270-8d40-7983096cdc81 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:58.505340 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d3c597-c844-4270-8d40-7983096cdc81 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:58.505615 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d3c597-c844-4270-8d40-7983096cdc81 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:58.516350 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d3c597-c844-4270-8d40-7983096cdc81 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:58.522749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-91d3c597-c844-4270-8d40-7983096cdc81 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 03:26:58.522749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:58.522749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:58.522749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:58.522749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:58.522749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:58.522749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:58.522749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:58.522749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:58.522749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:58.522749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:58.522749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:58.522749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:58.522749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:58.522749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:58.522749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:58.523729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:58.523729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:58.523729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:58.523729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:58.523729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:58.523729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:58.523729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:58.523729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:58.523729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:58.523729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:58.523729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:58.523729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:58.523729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:58.523729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:58.523729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:58.523729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:58.524528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:58.524528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:58.524528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:58.524528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:58.524528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:58.524528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:58.524528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:58.524528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:58.524528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:58.524528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:58.524528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 03:26:58.524528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:26:58.524528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 133/276] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:58 2022] GET /identity/v3/roles/reader => generated 84 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:58.525371 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d3c597-c844-4270-8d40-7983096cdc81 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:58.537916 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceadf3ad-3846-48fb-9878-ba03285a1c5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:58.539381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceadf3ad-3846-48fb-9878-ba03285a1c5e None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:58.539842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceadf3ad-3846-48fb-9878-ba03285a1c5e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:58.540115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceadf3ad-3846-48fb-9878-ba03285a1c5e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:58.552059 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceadf3ad-3846-48fb-9878-ba03285a1c5e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:58.556925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-ceadf3ad-3846-48fb-9878-ba03285a1c5e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:26:58.556925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:58.556925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:58.556925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:58.556925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:58.556925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:58.556925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:58.556925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:58.556925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:58.556925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:58.556925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:58.556925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:58.556925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:58.556925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:58.556925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:58.556925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:58.557844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:58.557844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:58.557844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:58.557844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:58.557844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:58.557844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:58.557844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:58.557844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:58.557844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:58.557844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:58.557844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:58.557844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:58.557844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:58.557844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:58.557844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:58.557844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:58.565357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:58.565357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:58.565357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:58.565357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:58.565357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:58.565357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:58.565357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:58.565357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:58.565357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:58.565357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:58.565357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:26:58.565357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:58.565357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 144/277] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:58 2022] GET /identity/v3/roles/member => generated 84 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:58.565357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceadf3ad-3846-48fb-9878-ba03285a1c5e None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:58.603156 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:58.652701 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d3c597-c844-4270-8d40-7983096cdc81 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:58.675738 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceadf3ad-3846-48fb-9878-ba03285a1c5e None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:58.702338 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb03ead3-5c1b-4404-a516-382b559bb90c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:58.703017 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb03ead3-5c1b-4404-a516-382b559bb90c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:58.703306 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb03ead3-5c1b-4404-a516-382b559bb90c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:58.703575 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb03ead3-5c1b-4404-a516-382b559bb90c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:58.705341 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb03ead3-5c1b-4404-a516-382b559bb90c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:58.706988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb03ead3-5c1b-4404-a516-382b559bb90c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:58.708912 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 134/278] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:26:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:58.709772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb03ead3-5c1b-4404-a516-382b559bb90c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:58.721554 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:58.729228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f8dfe8-e81b-486c-9cec-d2a2dc040768 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:58.730708 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f8dfe8-e81b-486c-9cec-d2a2dc040768 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:58.742123 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f8dfe8-e81b-486c-9cec-d2a2dc040768 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:58.742123 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f8dfe8-e81b-486c-9cec-d2a2dc040768 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:58.742123 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f8dfe8-e81b-486c-9cec-d2a2dc040768 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:58.742123 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f8dfe8-e81b-486c-9cec-d2a2dc040768 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:58.742123 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f8dfe8-e81b-486c-9cec-d2a2dc040768 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:58.747073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 145/279] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:26:58 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:58.747073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f8dfe8-e81b-486c-9cec-d2a2dc040768 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:58.831235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb03ead3-5c1b-4404-a516-382b559bb90c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:58.872908 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4cd057-adb5-45d0-83f9-c6558f346080 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:58.873817 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4cd057-adb5-45d0-83f9-c6558f346080 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:58.874235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4cd057-adb5-45d0-83f9-c6558f346080 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:58.874679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4cd057-adb5-45d0-83f9-c6558f346080 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:58.875611 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4cd057-adb5-45d0-83f9-c6558f346080 None admin] RBAC: Adding query filter params (name=member) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:58.875922 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4cd057-adb5-45d0-83f9-c6558f346080 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:58.878083 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:58.878239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4cd057-adb5-45d0-83f9-c6558f346080 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:58.884971 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f8dfe8-e81b-486c-9cec-d2a2dc040768 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:58.888339 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 135/280] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:26:58 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:58.892762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4cd057-adb5-45d0-83f9-c6558f346080 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:58.926054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd4ea79-df69-440d-ab08-6eee22918a6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:58.926636 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd4ea79-df69-440d-ab08-6eee22918a6c None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:58.927033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd4ea79-df69-440d-ab08-6eee22918a6c None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:58.927300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd4ea79-df69-440d-ab08-6eee22918a6c None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:58.927849 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd4ea79-df69-440d-ab08-6eee22918a6c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:58.929530 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd4ea79-df69-440d-ab08-6eee22918a6c None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:58.968653 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:58.997833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4cd057-adb5-45d0-83f9-c6558f346080 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:59.030476 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0559b237-9028-49fc-b2be-6534d97f509c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:59.031124 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0559b237-9028-49fc-b2be-6534d97f509c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:59.031409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0559b237-9028-49fc-b2be-6534d97f509c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:59.031669 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0559b237-9028-49fc-b2be-6534d97f509c None admin] PATH_INFO: `/v3/users/38a0fa7debad4252bd89cf6377b57c5c` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:59.032803 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0559b237-9028-49fc-b2be-6534d97f509c None admin] RBAC: Authorizing `identity:get_user(user_id=38a0fa7debad4252bd89cf6377b57c5c)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:59.034278 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0559b237-9028-49fc-b2be-6534d97f509c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:59.035858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 136/281] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:58 2022] GET /identity/v3/users/38a0fa7debad4252bd89cf6377b57c5c => generated 297 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:59.036601 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0559b237-9028-49fc-b2be-6534d97f509c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:59.066206 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 146/282] 10.222.0.49 () {60 vars in 1132 bytes} [Fri Sep 16 03:26:58 2022] POST /identity/v3/users => generated 293 bytes in 321 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:26:59.067627 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a4dfaa-bce4-411d-95a5-8ec41596b668 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:59.067917 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a4dfaa-bce4-411d-95a5-8ec41596b668 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:59.068430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a4dfaa-bce4-411d-95a5-8ec41596b668 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:59.071003 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 147/283] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:26:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:26:59.071003 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a4dfaa-bce4-411d-95a5-8ec41596b668 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:59.150776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0559b237-9028-49fc-b2be-6534d97f509c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:59.192838 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6bce89-7af7-4851-83fd-bb88b3080f6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:59.193060 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a4dfaa-bce4-411d-95a5-8ec41596b668 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:59.193645 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6bce89-7af7-4851-83fd-bb88b3080f6f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:59.193928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6bce89-7af7-4851-83fd-bb88b3080f6f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:59.194202 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6bce89-7af7-4851-83fd-bb88b3080f6f None admin] PATH_INFO: `/v3/users/ca78248c24024d6d85415ebaf31ca56e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:59.206143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:59.212144 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6bce89-7af7-4851-83fd-bb88b3080f6f None admin] RBAC: Authorizing `identity:get_user(user_id=ca78248c24024d6d85415ebaf31ca56e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:59.213654 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6bce89-7af7-4851-83fd-bb88b3080f6f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:59.215602 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 137/284] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:26:59 2022] GET /identity/v3/users/ca78248c24024d6d85415ebaf31ca56e => generated 293 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:59.216792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd66f98e-6e8a-4e95-bf88-39367ac4714a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:59.218949 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd66f98e-6e8a-4e95-bf88-39367ac4714a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:59.218949 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd66f98e-6e8a-4e95-bf88-39367ac4714a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:59.237174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d22397e-eab5-4dfc-ae73-36b76c1f31fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:59.237392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-dd66f98e-6e8a-4e95-bf88-39367ac4714a None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:26:59.237822 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d22397e-eab5-4dfc-ae73-36b76c1f31fb None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:59.238100 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d22397e-eab5-4dfc-ae73-36b76c1f31fb None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:59.238374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d22397e-eab5-4dfc-ae73-36b76c1f31fb None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:59.239511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d22397e-eab5-4dfc-ae73-36b76c1f31fb None admin] RBAC: Authorizing `identity:get_project(project_id=5f8ae3affd36481c8ce85ec30e7dedd9)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:59.239615 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-dd66f98e-6e8a-4e95-bf88-39367ac4714a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:26:59.240955 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d22397e-eab5-4dfc-ae73-36b76c1f31fb None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:59.243283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 148/285] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:26:59 2022] GET /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9 => generated 302 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:59.243401 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d22397e-eab5-4dfc-ae73-36b76c1f31fb None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:59.268303 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:59.326462 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 138/286] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:26:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:26:59.326462 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd66f98e-6e8a-4e95-bf88-39367ac4714a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:59.365601 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d22397e-eab5-4dfc-ae73-36b76c1f31fb None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:59.407240 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481e4c88-5ab6-4965-8b89-1122a7f1f2bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:59.408032 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481e4c88-5ab6-4965-8b89-1122a7f1f2bd None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:59.408395 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481e4c88-5ab6-4965-8b89-1122a7f1f2bd None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:59.408677 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481e4c88-5ab6-4965-8b89-1122a7f1f2bd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:59.409460 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481e4c88-5ab6-4965-8b89-1122a7f1f2bd None admin] RBAC: Adding query filter params (role.id=589e1f1333a042b7a00c53384a2f5426, user.id=ca78248c24024d6d85415ebaf31ca56e, scope.system=all) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:26:59.409801 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481e4c88-5ab6-4965-8b89-1122a7f1f2bd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:59.412196 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481e4c88-5ab6-4965-8b89-1122a7f1f2bd None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:59.418072 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 149/287] 10.222.0.49 () {56 vars in 1308 bytes} [Fri Sep 16 03:26:59 2022] GET /identity/v3/role_assignments?user.id=ca78248c24024d6d85415ebaf31ca56e&scope.system=all&role.id=589e1f1333a042b7a00c53384a2f5426 => generated 226 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:26:59.457594 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd66f98e-6e8a-4e95-bf88-39367ac4714a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:59.490229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:59.500401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db516d2a-00a3-42ee-b753-700bb3c66e5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:59.501993 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db516d2a-00a3-42ee-b753-700bb3c66e5b None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:59.502657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db516d2a-00a3-42ee-b753-700bb3c66e5b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:59.503224 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db516d2a-00a3-42ee-b753-700bb3c66e5b None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9/users/38a0fa7debad4252bd89cf6377b57c5c/roles/363c2654ab4e456b8b9b0b8c8c28911e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:59.505708 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db516d2a-00a3-42ee-b753-700bb3c66e5b None admin] RBAC: Authorizing `identity:create_grant(project_id=5f8ae3affd36481c8ce85ec30e7dedd9, user_id=38a0fa7debad4252bd89cf6377b57c5c, role_id=363c2654ab4e456b8b9b0b8c8c28911e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:26:59.507513 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db516d2a-00a3-42ee-b753-700bb3c66e5b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:26:59.586652 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 139/288] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:26:59 2022] PUT /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9/users/38a0fa7debad4252bd89cf6377b57c5c/roles/363c2654ab4e456b8b9b0b8c8c28911e => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:26:59.632969 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481e4c88-5ab6-4965-8b89-1122a7f1f2bd None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:59.738191 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:26:59.757991 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481e4c88-5ab6-4965-8b89-1122a7f1f2bd None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:26:59.802685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01106ec-deb8-481b-ae27-486acb13e4e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:26:59.803380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01106ec-deb8-481b-ae27-486acb13e4e4 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:26:59.803666 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01106ec-deb8-481b-ae27-486acb13e4e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:26:59.803946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01106ec-deb8-481b-ae27-486acb13e4e4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:26:59.819440 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01106ec-deb8-481b-ae27-486acb13e4e4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:26:59.833273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-f01106ec-deb8-481b-ae27-486acb13e4e4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:26:59.833273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:26:59.833273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:26:59.833273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:26:59.833273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:26:59.833273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:26:59.833273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:26:59.833273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:26:59.833273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:26:59.833273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:26:59.833273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:26:59.833273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:26:59.833273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:26:59.833273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:26:59.833273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:26:59.833273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:26:59.834320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:26:59.834320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:26:59.834320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:26:59.834320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:26:59.834320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:26:59.834320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:26:59.834320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:26:59.834320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:26:59.834320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:26:59.834320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:26:59.834320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:26:59.834320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:26:59.834320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:26:59.834320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:26:59.834320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:26:59.834320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:26:59.835152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:26:59.835152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:26:59.835152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:26:59.835152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:26:59.835152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:26:59.835152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:26:59.835152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:26:59.835152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:26:59.835152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:26:59.835152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:26:59.835152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:26:59.835152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:26:59.837585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 150/289] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:26:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:26:59.844090 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db516d2a-00a3-42ee-b753-700bb3c66e5b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:26:59.968605 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db516d2a-00a3-42ee-b753-700bb3c66e5b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:00.006620 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f709f805-c268-4c16-bed4-0dc9651bfba3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:00.007199 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f709f805-c268-4c16-bed4-0dc9651bfba3 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:00.007483 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f709f805-c268-4c16-bed4-0dc9651bfba3 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:00.007760 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f709f805-c268-4c16-bed4-0dc9651bfba3 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:00.008430 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f709f805-c268-4c16-bed4-0dc9651bfba3 None admin] RBAC: Adding query filter params (name=member) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:00.008735 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f709f805-c268-4c16-bed4-0dc9651bfba3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:00.010126 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f709f805-c268-4c16-bed4-0dc9651bfba3 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:00.020097 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 140/290] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:26:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:00.025859 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01106ec-deb8-481b-ae27-486acb13e4e4 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:00.175852 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01106ec-deb8-481b-ae27-486acb13e4e4 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:00.214997 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de5537f-5b90-4a7d-857d-a174cbfd8719 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:00.215639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de5537f-5b90-4a7d-857d-a174cbfd8719 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:00.215917 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de5537f-5b90-4a7d-857d-a174cbfd8719 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:00.216187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de5537f-5b90-4a7d-857d-a174cbfd8719 None admin] PATH_INFO: `/v3/users/09371fdca3714892af909c965686fcaf` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:00.219600 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de5537f-5b90-4a7d-857d-a174cbfd8719 None admin] RBAC: Authorizing `identity:get_user(user_id=09371fdca3714892af909c965686fcaf)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:00.221290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de5537f-5b90-4a7d-857d-a174cbfd8719 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:00.223064 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 151/291] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:27:00 2022] GET /identity/v3/users/09371fdca3714892af909c965686fcaf => generated 297 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:00.228754 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f709f805-c268-4c16-bed4-0dc9651bfba3 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:00.314590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:00.387889 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f709f805-c268-4c16-bed4-0dc9651bfba3 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:00.444522 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a41935-be2a-4be5-9065-5d500aabdd86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:00.445153 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a41935-be2a-4be5-9065-5d500aabdd86 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:00.445435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a41935-be2a-4be5-9065-5d500aabdd86 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:00.445713 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a41935-be2a-4be5-9065-5d500aabdd86 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:00.541304 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a41935-be2a-4be5-9065-5d500aabdd86 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:00.542788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a41935-be2a-4be5-9065-5d500aabdd86 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:00.611785 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:00.676808 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 141/292] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 448 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:27:00.682038 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de5537f-5b90-4a7d-857d-a174cbfd8719 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:00.712839 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:00.833910 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de5537f-5b90-4a7d-857d-a174cbfd8719 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:00.882049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b7e7e6-2915-45ad-b020-bc7af2adaddb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:00.882610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b7e7e6-2915-45ad-b020-bc7af2adaddb None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:00.882895 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b7e7e6-2915-45ad-b020-bc7af2adaddb None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:00.883200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b7e7e6-2915-45ad-b020-bc7af2adaddb None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:00.884349 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b7e7e6-2915-45ad-b020-bc7af2adaddb None admin] RBAC: Authorizing `identity:get_project(project_id=5f8ae3affd36481c8ce85ec30e7dedd9)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:00.885804 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b7e7e6-2915-45ad-b020-bc7af2adaddb None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:00.887758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 152/293] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:27:00 2022] GET /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9 => generated 302 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:00.894336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a41935-be2a-4be5-9065-5d500aabdd86 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:01.049648 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a41935-be2a-4be5-9065-5d500aabdd86 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:01.096822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08de8eb7-1b58-4ef2-9712-0b90092a756c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:01.097429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08de8eb7-1b58-4ef2-9712-0b90092a756c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:01.097708 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08de8eb7-1b58-4ef2-9712-0b90092a756c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:01.098118 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08de8eb7-1b58-4ef2-9712-0b90092a756c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:01.098848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08de8eb7-1b58-4ef2-9712-0b90092a756c None admin] RBAC: Adding query filter params (role.id=589e1f1333a042b7a00c53384a2f5426, scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9, user.id=09371fdca3714892af909c965686fcaf) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:01.099181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08de8eb7-1b58-4ef2-9712-0b90092a756c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:01.100601 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08de8eb7-1b58-4ef2-9712-0b90092a756c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:01.112021 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 142/294] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:27:00 2022] GET /identity/v3/role_assignments?user.id=09371fdca3714892af909c965686fcaf&scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9&role.id=589e1f1333a042b7a00c53384a2f5426 => generated 608 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:01.908875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076faf00-b732-44f0-b30e-5334fb18994f None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:01.909407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076faf00-b732-44f0-b30e-5334fb18994f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:01.909407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076faf00-b732-44f0-b30e-5334fb18994f None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:01.911154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 153/295] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:01.916619 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc13be84-2043-4eb6-b948-340d8cdf5673 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:01.917114 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc13be84-2043-4eb6-b948-340d8cdf5673 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:01.917220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc13be84-2043-4eb6-b948-340d8cdf5673 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:01.938938 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-fc13be84-2043-4eb6-b948-340d8cdf5673 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:01.940196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fc13be84-2043-4eb6-b948-340d8cdf5673 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:01.948066 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:02.014070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 143/296] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:02.021688 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cf6937-1520-4d7b-8ac6-6857b3aa5f55 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:02.021975 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cf6937-1520-4d7b-8ac6-6857b3aa5f55 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:02.022250 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cf6937-1520-4d7b-8ac6-6857b3aa5f55 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:02.023841 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 154/297] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:02.028903 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f73e1f4-400b-48b9-ad75-839b2b7cd841 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:02.029443 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f73e1f4-400b-48b9-ad75-839b2b7cd841 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:02.029716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f73e1f4-400b-48b9-ad75-839b2b7cd841 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:02.048696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1f73e1f4-400b-48b9-ad75-839b2b7cd841 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:02.049946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1f73e1f4-400b-48b9-ad75-839b2b7cd841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:02.092964 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d707dc-0178-4346-b31c-4c209ac029ed None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:02.093289 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d707dc-0178-4346-b31c-4c209ac029ed None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:02.093525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d707dc-0178-4346-b31c-4c209ac029ed None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:02.095696 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 155/298] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:27:02.101583 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7546e9c7-66e0-4024-a659-815a65753878 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:02.102223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7546e9c7-66e0-4024-a659-815a65753878 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:02.102706 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7546e9c7-66e0-4024-a659-815a65753878 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:02.122402 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-7546e9c7-66e0-4024-a659-815a65753878 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:02.123938 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 144/299] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:02.124228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-7546e9c7-66e0-4024-a659-815a65753878 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:02.194173 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94085054-3e50-4f2f-ae44-1b1dc52b6cf9 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:02.194473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94085054-3e50-4f2f-ae44-1b1dc52b6cf9 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:02.194741 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94085054-3e50-4f2f-ae44-1b1dc52b6cf9 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:02.196367 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 145/300] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:02.197352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 156/301] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:02.201287 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5af5fc-1dad-4713-aa11-52cdc1e5e927 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:02.201571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5af5fc-1dad-4713-aa11-52cdc1e5e927 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:02.201847 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5af5fc-1dad-4713-aa11-52cdc1e5e927 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:02.220633 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-da5af5fc-1dad-4713-aa11-52cdc1e5e927 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:02.221797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-da5af5fc-1dad-4713-aa11-52cdc1e5e927 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:02.293886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7546e9c7-66e0-4024-a659-815a65753878 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:02.296777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 146/302] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:02.307445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:02.401950 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7546e9c7-66e0-4024-a659-815a65753878 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:02.421839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5af5fc-1dad-4713-aa11-52cdc1e5e927 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:02.425294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:02.437714 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17b8027-5ead-4ba8-b305-227c6f7c4e29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:02.438306 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17b8027-5ead-4ba8-b305-227c6f7c4e29 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:02.438695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17b8027-5ead-4ba8-b305-227c6f7c4e29 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:02.439000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17b8027-5ead-4ba8-b305-227c6f7c4e29 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:02.440179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17b8027-5ead-4ba8-b305-227c6f7c4e29 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:02.441635 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17b8027-5ead-4ba8-b305-227c6f7c4e29 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:02.443540 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 157/303] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:02 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:02.448734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17b8027-5ead-4ba8-b305-227c6f7c4e29 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:02.537821 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:02.541263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5af5fc-1dad-4713-aa11-52cdc1e5e927 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:02.562267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17b8027-5ead-4ba8-b305-227c6f7c4e29 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:02.580485 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada2e233-0057-4383-a4a0-e2d41cfae6d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:02.581134 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada2e233-0057-4383-a4a0-e2d41cfae6d6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:02.581419 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada2e233-0057-4383-a4a0-e2d41cfae6d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:02.581686 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada2e233-0057-4383-a4a0-e2d41cfae6d6 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:02.592791 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada2e233-0057-4383-a4a0-e2d41cfae6d6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:02.598055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-ada2e233-0057-4383-a4a0-e2d41cfae6d6 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 03:27:02.598055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:02.598055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:02.598055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:02.598055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:02.598055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:02.598055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:02.598055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:02.598055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:02.598055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:02.598055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:02.598055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:02.598055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:02.598055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:02.598055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:02.598055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:02.599147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:02.599147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:02.599147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:02.599147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:02.599147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:02.599147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:02.599147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:02.599147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:02.599147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:02.599147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:02.599147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:02.599147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:02.599147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:02.599147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:02.599147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:02.599147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:02.600008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:02.600008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:02.600008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:02.600008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:02.600008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:02.600008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:02.600008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:02.600008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:02.600008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:02.600008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:02.600008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 03:27:02.600008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:02.600008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 147/304] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:02 2022] GET /identity/v3/roles/reader => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:02.600692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada2e233-0057-4383-a4a0-e2d41cfae6d6 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:02.602673 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c881fd8-3ebf-49a6-bcd2-4501629b9c6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:02.603539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c881fd8-3ebf-49a6-bcd2-4501629b9c6a None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:02.603824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c881fd8-3ebf-49a6-bcd2-4501629b9c6a None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:02.604959 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c881fd8-3ebf-49a6-bcd2-4501629b9c6a None admin] PATH_INFO: `/v3/groups` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:02.605856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c881fd8-3ebf-49a6-bcd2-4501629b9c6a None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:02.607357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c881fd8-3ebf-49a6-bcd2-4501629b9c6a None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:02.660665 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:02.696718 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada2e233-0057-4383-a4a0-e2d41cfae6d6 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:02.702575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 158/305] 10.222.0.49 () {60 vars in 1134 bytes} [Fri Sep 16 03:27:02 2022] POST /identity/v3/groups => generated 229 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:27:02.703365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c881fd8-3ebf-49a6-bcd2-4501629b9c6a None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:02.737498 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0867faaf-b26f-4aa6-aec8-6181b9a59f14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:02.738696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0867faaf-b26f-4aa6-aec8-6181b9a59f14 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:02.739172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0867faaf-b26f-4aa6-aec8-6181b9a59f14 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:02.739594 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0867faaf-b26f-4aa6-aec8-6181b9a59f14 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:02.752297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0867faaf-b26f-4aa6-aec8-6181b9a59f14 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:02.758551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-0867faaf-b26f-4aa6-aec8-6181b9a59f14 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:27:02.758551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:02.758551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:02.758551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:02.758551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:02.758551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:02.758551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:02.758551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:02.758551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:02.758551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:02.758551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:02.758551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:02.758551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:02.758551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:02.758551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:02.758551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:02.759726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:02.759726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:02.759726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:02.759726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:02.759726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:02.759726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:02.759726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:02.759726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:02.759726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:02.759726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:02.759726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:02.759726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:02.759726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:02.759726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:02.759726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:02.759726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:02.760724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:02.760724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:02.760724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:02.760724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:02.760724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:02.760724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:02.760724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:02.760724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:02.760724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:02.760724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:02.760724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:27:02.760724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:02.762793 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 148/306] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:02.768336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0867faaf-b26f-4aa6-aec8-6181b9a59f14 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:02.834099 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c881fd8-3ebf-49a6-bcd2-4501629b9c6a None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:02.872989 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47273f7-359a-4654-a898-922be0b0d6cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:02.873542 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47273f7-359a-4654-a898-922be0b0d6cc None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:02.873821 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47273f7-359a-4654-a898-922be0b0d6cc None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:02.874096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47273f7-359a-4654-a898-922be0b0d6cc None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:02.881229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0867faaf-b26f-4aa6-aec8-6181b9a59f14 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:02.883262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:02.887810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47273f7-359a-4654-a898-922be0b0d6cc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:02.894213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-c47273f7-359a-4654-a898-922be0b0d6cc None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 03:27:02.894213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:02.894213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:02.894213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:02.894213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:02.894213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:02.894213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:02.894213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:02.894213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:02.894213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:02.894213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:02.894213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:02.894213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:02.894213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:02.894213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:02.894213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:02.899761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:02.899761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:02.899761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:02.899761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:02.899761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:02.899761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:02.899761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:02.899761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:02.899761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:02.899761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:02.899761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:02.899761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:02.899761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:02.899761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:02.899761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:02.899761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:02.900585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:02.900585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:02.900585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:02.900585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:02.900585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:02.900585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:02.900585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:02.900585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:02.900585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:02.900585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:02.900585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 03:27:02.900585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:02.900585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 159/307] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:02 2022] GET /identity/v3/roles/reader => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:02.900585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47273f7-359a-4654-a898-922be0b0d6cc None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:02.913905 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:02.923118 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82f7bf2-b9e1-4fe9-90cc-3c05258a47ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:02.923716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82f7bf2-b9e1-4fe9-90cc-3c05258a47ca None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:02.924040 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82f7bf2-b9e1-4fe9-90cc-3c05258a47ca None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:02.924303 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82f7bf2-b9e1-4fe9-90cc-3c05258a47ca None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:02.924934 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82f7bf2-b9e1-4fe9-90cc-3c05258a47ca None admin] RBAC: Adding query filter params (name=reader) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:02.925237 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82f7bf2-b9e1-4fe9-90cc-3c05258a47ca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:02.926630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82f7bf2-b9e1-4fe9-90cc-3c05258a47ca None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:02.935179 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 149/308] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:27:02 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:02.940551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82f7bf2-b9e1-4fe9-90cc-3c05258a47ca None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:03.034005 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47273f7-359a-4654-a898-922be0b0d6cc None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:03.041465 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82f7bf2-b9e1-4fe9-90cc-3c05258a47ca None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:03.070179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430275ad-d029-4a46-8bbd-3493c66218cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:03.070786 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430275ad-d029-4a46-8bbd-3493c66218cd None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:03.071090 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430275ad-d029-4a46-8bbd-3493c66218cd None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:03.071363 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430275ad-d029-4a46-8bbd-3493c66218cd None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:03.072015 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430275ad-d029-4a46-8bbd-3493c66218cd None admin] RBAC: Adding query filter params (name=member) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:03.072315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430275ad-d029-4a46-8bbd-3493c66218cd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:03.073697 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430275ad-d029-4a46-8bbd-3493c66218cd None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:03.080458 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be30b6a3-430c-4633-8ede-c0e066e7c332 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:03.081077 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be30b6a3-430c-4633-8ede-c0e066e7c332 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:03.081357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be30b6a3-430c-4633-8ede-c0e066e7c332 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:03.081630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be30b6a3-430c-4633-8ede-c0e066e7c332 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:03.086384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 160/309] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:27:02 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:03.086506 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be30b6a3-430c-4633-8ede-c0e066e7c332 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:03.086506 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be30b6a3-430c-4633-8ede-c0e066e7c332 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:03.086659 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430275ad-d029-4a46-8bbd-3493c66218cd None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:03.086727 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be30b6a3-430c-4633-8ede-c0e066e7c332 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:03.094486 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 150/310] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:27:02 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:03.095880 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be30b6a3-430c-4633-8ede-c0e066e7c332 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:03.147316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:03.155091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:03.196498 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430275ad-d029-4a46-8bbd-3493c66218cd None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:03.203433 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be30b6a3-430c-4633-8ede-c0e066e7c332 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:03.234326 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8521de29-9bf5-4e91-a31c-a2d380172233 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:03.234991 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8521de29-9bf5-4e91-a31c-a2d380172233 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:03.235274 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8521de29-9bf5-4e91-a31c-a2d380172233 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:03.236069 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8521de29-9bf5-4e91-a31c-a2d380172233 None admin] PATH_INFO: `/v3/users/3af08fdc88fd48fe91b3e2370bbe8745` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:03.241764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5a9774-5e13-4f1c-8c16-ae1253828292 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:03.242397 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a9774-5e13-4f1c-8c16-ae1253828292 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:03.242680 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a9774-5e13-4f1c-8c16-ae1253828292 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:03.242982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a9774-5e13-4f1c-8c16-ae1253828292 None admin] PATH_INFO: `/v3/users/ca78248c24024d6d85415ebaf31ca56e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:03.244113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5a9774-5e13-4f1c-8c16-ae1253828292 None admin] RBAC: Authorizing `identity:get_user(user_id=ca78248c24024d6d85415ebaf31ca56e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:03.245539 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5a9774-5e13-4f1c-8c16-ae1253828292 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:03.247455 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 151/311] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:27:03 2022] GET /identity/v3/users/ca78248c24024d6d85415ebaf31ca56e => generated 293 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:03.248218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5a9774-5e13-4f1c-8c16-ae1253828292 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:03.254421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8521de29-9bf5-4e91-a31c-a2d380172233 None admin] RBAC: Authorizing `identity:get_user(user_id=3af08fdc88fd48fe91b3e2370bbe8745)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:03.256116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8521de29-9bf5-4e91-a31c-a2d380172233 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:03.257930 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 161/312] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:27:03 2022] GET /identity/v3/users/3af08fdc88fd48fe91b3e2370bbe8745 => generated 293 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:03.258881 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8521de29-9bf5-4e91-a31c-a2d380172233 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:03.360260 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5a9774-5e13-4f1c-8c16-ae1253828292 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:03.369778 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8521de29-9bf5-4e91-a31c-a2d380172233 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:03.376237 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:03.387171 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:03.397979 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc91818-5674-4237-b43e-2f2863bf9102 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:03.398674 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc91818-5674-4237-b43e-2f2863bf9102 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:03.398955 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc91818-5674-4237-b43e-2f2863bf9102 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:03.399230 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc91818-5674-4237-b43e-2f2863bf9102 None admin] PATH_INFO: `/v3/users/38a0fa7debad4252bd89cf6377b57c5c` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:03.400373 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc91818-5674-4237-b43e-2f2863bf9102 None admin] RBAC: Authorizing `identity:get_user(user_id=38a0fa7debad4252bd89cf6377b57c5c)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:03.408397 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc91818-5674-4237-b43e-2f2863bf9102 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:03.408397 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 152/313] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:27:03 2022] GET /identity/v3/users/38a0fa7debad4252bd89cf6377b57c5c => generated 297 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:03.408397 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc91818-5674-4237-b43e-2f2863bf9102 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:03.410248 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a920643c-a54f-48ae-9cc1-b5df588c71bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:03.411040 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a920643c-a54f-48ae-9cc1-b5df588c71bf None admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:03.411383 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a920643c-a54f-48ae-9cc1-b5df588c71bf None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:03.411726 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a920643c-a54f-48ae-9cc1-b5df588c71bf None admin] PATH_INFO: `/v3/system/users/ca78248c24024d6d85415ebaf31ca56e/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:03.413218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a920643c-a54f-48ae-9cc1-b5df588c71bf None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=ca78248c24024d6d85415ebaf31ca56e, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:03.414683 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a920643c-a54f-48ae-9cc1-b5df588c71bf None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:03.492126 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 162/314] 10.222.0.49 () {58 vars in 1334 bytes} [Fri Sep 16 03:27:03 2022] PUT /identity/v3/system/users/ca78248c24024d6d85415ebaf31ca56e/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:27:03.495922 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a920643c-a54f-48ae-9cc1-b5df588c71bf None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:03.519331 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc91818-5674-4237-b43e-2f2863bf9102 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:03.575135 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f33786-3139-40d2-8f03-6f8507d58528 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:03.576143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f33786-3139-40d2-8f03-6f8507d58528 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:03.576627 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f33786-3139-40d2-8f03-6f8507d58528 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:03.577450 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f33786-3139-40d2-8f03-6f8507d58528 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:03.579575 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f33786-3139-40d2-8f03-6f8507d58528 None admin] RBAC: Adding query filter params (role.id=363c2654ab4e456b8b9b0b8c8c28911e, user.id=3af08fdc88fd48fe91b3e2370bbe8745, scope.system=all) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:03.580516 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f33786-3139-40d2-8f03-6f8507d58528 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:03.582269 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f33786-3139-40d2-8f03-6f8507d58528 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:03.590686 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 153/315] 10.222.0.49 () {56 vars in 1308 bytes} [Fri Sep 16 03:27:03 2022] GET /identity/v3/role_assignments?user.id=3af08fdc88fd48fe91b3e2370bbe8745&scope.system=all&role.id=363c2654ab4e456b8b9b0b8c8c28911e => generated 226 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:03.627057 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a920643c-a54f-48ae-9cc1-b5df588c71bf None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:03.669344 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30bea2c-e586-4c6d-8723-1829b44d89b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:03.669980 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30bea2c-e586-4c6d-8723-1829b44d89b5 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:03.670259 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30bea2c-e586-4c6d-8723-1829b44d89b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:03.670530 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30bea2c-e586-4c6d-8723-1829b44d89b5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:03.728936 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:03.749621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30bea2c-e586-4c6d-8723-1829b44d89b5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:03.751245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30bea2c-e586-4c6d-8723-1829b44d89b5 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:03.863051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 163/316] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:27:03.871398 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f33786-3139-40d2-8f03-6f8507d58528 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:03.922328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:03.978413 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f33786-3139-40d2-8f03-6f8507d58528 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:04.049683 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e2ae46-9586-4836-8e29-e436894a5943 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:04.050320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e2ae46-9586-4836-8e29-e436894a5943 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:04.050598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e2ae46-9586-4836-8e29-e436894a5943 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:04.050869 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e2ae46-9586-4836-8e29-e436894a5943 None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:04.052031 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e2ae46-9586-4836-8e29-e436894a5943 None admin] RBAC: Authorizing `identity:get_project(project_id=5f8ae3affd36481c8ce85ec30e7dedd9)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:04.053579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e2ae46-9586-4836-8e29-e436894a5943 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:04.055529 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 154/317] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:27:03 2022] GET /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9 => generated 302 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:04.064078 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30bea2c-e586-4c6d-8723-1829b44d89b5 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:04.148135 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:04.167699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30bea2c-e586-4c6d-8723-1829b44d89b5 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:04.206058 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9524ee-ac45-4c44-af06-e8890935aa8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:04.206726 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9524ee-ac45-4c44-af06-e8890935aa8b None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:04.207034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9524ee-ac45-4c44-af06-e8890935aa8b None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:04.207308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9524ee-ac45-4c44-af06-e8890935aa8b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:04.208045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9524ee-ac45-4c44-af06-e8890935aa8b None admin] RBAC: Adding query filter params (role.id=363c2654ab4e456b8b9b0b8c8c28911e, scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9, user.id=38a0fa7debad4252bd89cf6377b57c5c) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:04.208349 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9524ee-ac45-4c44-af06-e8890935aa8b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:04.209793 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9524ee-ac45-4c44-af06-e8890935aa8b None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:04.216082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 164/318] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:27:04 2022] GET /identity/v3/role_assignments?user.id=38a0fa7debad4252bd89cf6377b57c5c&scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9&role.id=363c2654ab4e456b8b9b0b8c8c28911e => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:04.937322 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c07335-1f68-4f67-8993-2500221dbb4f None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:04.937725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c07335-1f68-4f67-8993-2500221dbb4f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:04.937902 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c07335-1f68-4f67-8993-2500221dbb4f None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:04.939737 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 155/319] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:04.944764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30be6b1-c803-4e31-acd0-5d5b61c809d1 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:04.945178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30be6b1-c803-4e31-acd0-5d5b61c809d1 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:04.945532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30be6b1-c803-4e31-acd0-5d5b61c809d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:04.964281 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-b30be6b1-c803-4e31-acd0-5d5b61c809d1 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:04.965460 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b30be6b1-c803-4e31-acd0-5d5b61c809d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:05.034581 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 165/320] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:05.312458 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c07335-1f68-4f67-8993-2500221dbb4f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:05.419418 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c07335-1f68-4f67-8993-2500221dbb4f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:05.433208 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:05.452841 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f337968-a691-491c-a357-d13418b5ad3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:05.453471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f337968-a691-491c-a357-d13418b5ad3a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:05.453745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f337968-a691-491c-a357-d13418b5ad3a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:05.454011 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f337968-a691-491c-a357-d13418b5ad3a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:05.455632 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f337968-a691-491c-a357-d13418b5ad3a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:05.457062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f337968-a691-491c-a357-d13418b5ad3a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:05.458768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 156/321] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:05.464053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30be6b1-c803-4e31-acd0-5d5b61c809d1 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:05.541579 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:05.564766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30be6b1-c803-4e31-acd0-5d5b61c809d1 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:05.600894 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3780268c-1d8b-4830-ad7e-ffdf57f77441 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:05.601520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3780268c-1d8b-4830-ad7e-ffdf57f77441 None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:05.601801 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3780268c-1d8b-4830-ad7e-ffdf57f77441 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:05.602073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3780268c-1d8b-4830-ad7e-ffdf57f77441 None admin] PATH_INFO: `/v3/groups` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:05.602711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3780268c-1d8b-4830-ad7e-ffdf57f77441 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:05.604372 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3780268c-1d8b-4830-ad7e-ffdf57f77441 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:05.701203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 166/322] 10.222.0.49 () {60 vars in 1134 bytes} [Fri Sep 16 03:27:05 2022] POST /identity/v3/groups => generated 226 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:27:05.784505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67b5684-3d30-41a8-9ef6-ac0a1d0458d6 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:05.784853 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67b5684-3d30-41a8-9ef6-ac0a1d0458d6 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:05.785065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67b5684-3d30-41a8-9ef6-ac0a1d0458d6 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:05.786633 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 157/323] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:05.792341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7288fa42-bdc2-4163-b518-bb4db2f31ab0 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:05.792625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7288fa42-bdc2-4163-b518-bb4db2f31ab0 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:05.792907 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7288fa42-bdc2-4163-b518-bb4db2f31ab0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:05.811898 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-7288fa42-bdc2-4163-b518-bb4db2f31ab0 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:05.813114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-7288fa42-bdc2-4163-b518-bb4db2f31ab0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:05.819691 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37420b70-7716-4b31-a2aa-8fe7d9d70808 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:05.819978 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37420b70-7716-4b31-a2aa-8fe7d9d70808 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:05.820170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37420b70-7716-4b31-a2aa-8fe7d9d70808 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:05.821550 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 158/324] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:05.827127 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f464c2f-a135-4dc6-84d8-42dd7abb0aca None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:05.827415 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f464c2f-a135-4dc6-84d8-42dd7abb0aca None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:05.827694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f464c2f-a135-4dc6-84d8-42dd7abb0aca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:05.848417 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1f464c2f-a135-4dc6-84d8-42dd7abb0aca None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:05.849640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1f464c2f-a135-4dc6-84d8-42dd7abb0aca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:05.895010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 167/325] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:05.937104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 159/326] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 03:27:06.173587 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7288fa42-bdc2-4163-b518-bb4db2f31ab0 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:06.206472 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f464c2f-a135-4dc6-84d8-42dd7abb0aca None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:06.248601 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:06.299591 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7288fa42-bdc2-4163-b518-bb4db2f31ab0 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:06.344552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f464c2f-a135-4dc6-84d8-42dd7abb0aca None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:06.348386 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f852fc-421a-46ef-a378-4b9a09db0068 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:06.348646 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f852fc-421a-46ef-a378-4b9a09db0068 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:06.348989 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f852fc-421a-46ef-a378-4b9a09db0068 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:06.349327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f852fc-421a-46ef-a378-4b9a09db0068 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:06.362953 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:06.364706 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f852fc-421a-46ef-a378-4b9a09db0068 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:06.371137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-37f852fc-421a-46ef-a378-4b9a09db0068 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:27:06.371137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:06.371137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:06.371137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:06.371137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:06.371137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:06.371137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:06.371137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:06.371137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:06.371137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:06.371137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:06.371137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:06.371137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:06.371137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:06.371137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:06.371137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:06.388926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:06.388926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:06.388926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:06.388926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:06.388926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:06.388926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:06.388926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:06.388926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:06.388926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:06.388926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:06.388926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:06.388926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:06.388926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:06.388926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:06.388926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:06.388926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:06.389958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:06.389958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:06.389958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:06.389958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:06.389958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:06.389958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:06.389958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:06.389958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:06.389958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:06.389958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:06.389958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:27:06.389958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:06.389958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 168/327] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:06 2022] GET /identity/v3/roles/member => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:06.389958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f852fc-421a-46ef-a378-4b9a09db0068 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:06.390878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3717f26-8333-47a9-bced-675135436e29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:06.390878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3717f26-8333-47a9-bced-675135436e29 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:06.390878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3717f26-8333-47a9-bced-675135436e29 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:06.390878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3717f26-8333-47a9-bced-675135436e29 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:06.402738 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3717f26-8333-47a9-bced-675135436e29 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:06.409745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-d3717f26-8333-47a9-bced-675135436e29 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 03:27:06.409745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:06.409745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:06.409745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:06.409745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:06.409745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:06.409745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:06.409745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:06.409745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:06.409745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:06.409745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:06.409745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:06.409745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:06.409745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:06.409745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:06.409745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:06.411681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:06.411681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:06.411681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:06.411681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:06.411681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:06.411681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:06.411681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:06.411681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:06.411681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:06.411681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:06.411681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:06.411681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:06.411681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:06.411681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:06.411681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:06.411681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:06.413438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:06.413438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:06.413438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:06.413438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:06.413438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:06.413438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:06.413438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:06.413438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:06.413438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:06.413438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:06.413438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 03:27:06.413438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:06.413438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 160/328] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:06 2022] GET /identity/v3/roles/reader => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:27:06.417915 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3717f26-8333-47a9-bced-675135436e29 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:06.528043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f852fc-421a-46ef-a378-4b9a09db0068 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:06.535113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3717f26-8333-47a9-bced-675135436e29 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:06.574457 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:06.589490 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9c129d-9932-418e-bd5f-196257096434 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:06.596483 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9c129d-9932-418e-bd5f-196257096434 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:06.596483 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9c129d-9932-418e-bd5f-196257096434 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:06.596483 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9c129d-9932-418e-bd5f-196257096434 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:06.596483 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9c129d-9932-418e-bd5f-196257096434 None admin] RBAC: Adding query filter params (name=member) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:06.596483 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9c129d-9932-418e-bd5f-196257096434 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:06.596483 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9c129d-9932-418e-bd5f-196257096434 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:06.602128 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea682ad4-c74f-41de-86d4-496f70e0ce31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:06.602812 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea682ad4-c74f-41de-86d4-496f70e0ce31 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:06.603119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea682ad4-c74f-41de-86d4-496f70e0ce31 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:06.603387 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea682ad4-c74f-41de-86d4-496f70e0ce31 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:06.604106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea682ad4-c74f-41de-86d4-496f70e0ce31 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:06.604474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea682ad4-c74f-41de-86d4-496f70e0ce31 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:06.605854 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea682ad4-c74f-41de-86d4-496f70e0ce31 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:06.616370 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 169/329] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:27:06 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:06.622406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9c129d-9932-418e-bd5f-196257096434 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:06.627113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 161/330] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:27:06 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:06.633794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea682ad4-c74f-41de-86d4-496f70e0ce31 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:06.717187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:06.765112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9c129d-9932-418e-bd5f-196257096434 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:06.806482 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea682ad4-c74f-41de-86d4-496f70e0ce31 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:06.844100 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ff7bc6-f47e-48fe-b10d-405f67e08d7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:06.844859 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ff7bc6-f47e-48fe-b10d-405f67e08d7d None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:06.845290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ff7bc6-f47e-48fe-b10d-405f67e08d7d None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:06.845615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ff7bc6-f47e-48fe-b10d-405f67e08d7d None admin] PATH_INFO: `/v3/users/ca78248c24024d6d85415ebaf31ca56e` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:06.846443 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996ddec0-1d91-4d2f-99b3-83a0e5adcf4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:06.847122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996ddec0-1d91-4d2f-99b3-83a0e5adcf4e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:06.847405 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996ddec0-1d91-4d2f-99b3-83a0e5adcf4e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:06.847684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996ddec0-1d91-4d2f-99b3-83a0e5adcf4e None admin] PATH_INFO: `/v3/users/3af08fdc88fd48fe91b3e2370bbe8745` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:06.847845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ff7bc6-f47e-48fe-b10d-405f67e08d7d None admin] RBAC: Authorizing `identity:get_user(user_id=ca78248c24024d6d85415ebaf31ca56e)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:06.848837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996ddec0-1d91-4d2f-99b3-83a0e5adcf4e None admin] RBAC: Authorizing `identity:get_user(user_id=3af08fdc88fd48fe91b3e2370bbe8745)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:06.849289 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ff7bc6-f47e-48fe-b10d-405f67e08d7d None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:06.850271 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996ddec0-1d91-4d2f-99b3-83a0e5adcf4e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:06.851068 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 170/331] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:27:06 2022] GET /identity/v3/users/ca78248c24024d6d85415ebaf31ca56e => generated 293 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:06.855042 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 162/332] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:27:06 2022] GET /identity/v3/users/3af08fdc88fd48fe91b3e2370bbe8745 => generated 293 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:06.857980 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ff7bc6-f47e-48fe-b10d-405f67e08d7d None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:06.861807 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996ddec0-1d91-4d2f-99b3-83a0e5adcf4e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:06.989373 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996ddec0-1d91-4d2f-99b3-83a0e5adcf4e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:06.990072 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:06.992713 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:07.061799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7ad463-67a2-4652-9fdf-da677312612c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:07.062388 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7ad463-67a2-4652-9fdf-da677312612c None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:07.062664 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7ad463-67a2-4652-9fdf-da677312612c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:07.062955 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7ad463-67a2-4652-9fdf-da677312612c None admin] PATH_INFO: `/v3/system/users/3af08fdc88fd48fe91b3e2370bbe8745/roles/363c2654ab4e456b8b9b0b8c8c28911e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:07.065396 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d7ad463-67a2-4652-9fdf-da677312612c None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=3af08fdc88fd48fe91b3e2370bbe8745, role_id=363c2654ab4e456b8b9b0b8c8c28911e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:07.066873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d7ad463-67a2-4652-9fdf-da677312612c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:07.077648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ff7bc6-f47e-48fe-b10d-405f67e08d7d None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:07.116996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84987166-7970-444b-b321-9337def55854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:07.117605 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84987166-7970-444b-b321-9337def55854 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:07.117938 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84987166-7970-444b-b321-9337def55854 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:07.118210 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84987166-7970-444b-b321-9337def55854 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:07.118963 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84987166-7970-444b-b321-9337def55854 None admin] RBAC: Adding query filter params (role.id=589e1f1333a042b7a00c53384a2f5426, user.id=ca78248c24024d6d85415ebaf31ca56e, scope.system=all) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:07.119335 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84987166-7970-444b-b321-9337def55854 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:07.120818 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84987166-7970-444b-b321-9337def55854 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:07.133806 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 171/333] 10.222.0.49 () {56 vars in 1308 bytes} [Fri Sep 16 03:27:06 2022] GET /identity/v3/role_assignments?user.id=ca78248c24024d6d85415ebaf31ca56e&scope.system=all&role.id=589e1f1333a042b7a00c53384a2f5426 => generated 510 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:07.138225 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 163/334] 10.222.0.49 () {58 vars in 1334 bytes} [Fri Sep 16 03:27:06 2022] PUT /identity/v3/system/users/3af08fdc88fd48fe91b3e2370bbe8745/roles/363c2654ab4e456b8b9b0b8c8c28911e => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:27:08.398745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a235861-f2fb-4a4d-af89-0a26a91ee8e2 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:08.399729 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a235861-f2fb-4a4d-af89-0a26a91ee8e2 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:08.400185 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a235861-f2fb-4a4d-af89-0a26a91ee8e2 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:08.401981 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 172/335] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:08.416712 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dbefe3-c341-4626-afb9-2428d4044a3f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:08.417662 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dbefe3-c341-4626-afb9-2428d4044a3f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:08.418140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dbefe3-c341-4626-afb9-2428d4044a3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:08.444129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-60dbefe3-c341-4626-afb9-2428d4044a3f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:08.445324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-60dbefe3-c341-4626-afb9-2428d4044a3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:08.535050 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 164/336] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:08.902090 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d839fa11-12fa-4403-84ad-2936035853b7 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:08.902797 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d839fa11-12fa-4403-84ad-2936035853b7 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:08.903305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d839fa11-12fa-4403-84ad-2936035853b7 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:08.905391 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 173/337] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:08.911353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081a7928-34a3-4287-b3ef-71f6cfe00969 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:08.911960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081a7928-34a3-4287-b3ef-71f6cfe00969 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:08.912434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081a7928-34a3-4287-b3ef-71f6cfe00969 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:08.936991 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-081a7928-34a3-4287-b3ef-71f6cfe00969 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:08.948130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-081a7928-34a3-4287-b3ef-71f6cfe00969 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:08.963291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac029c4-a034-4dda-8a65-c1fe5cb13083 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:08.963886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac029c4-a034-4dda-8a65-c1fe5cb13083 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:08.964362 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac029c4-a034-4dda-8a65-c1fe5cb13083 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:08.966093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 174/338] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:08.972561 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc959865-2a04-4341-b013-a1de36784cad None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:08.972830 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc959865-2a04-4341-b013-a1de36784cad None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:08.973098 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc959865-2a04-4341-b013-a1de36784cad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:09.011936 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-fc959865-2a04-4341-b013-a1de36784cad None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:09.013153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-fc959865-2a04-4341-b013-a1de36784cad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:09.043007 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 165/339] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:09.044003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081a7928-34a3-4287-b3ef-71f6cfe00969 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:09.074120 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:09.158791 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081a7928-34a3-4287-b3ef-71f6cfe00969 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:09.176694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 175/340] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:09.194705 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbdc8c5-c678-4884-abe1-f02a76b83748 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:09.195386 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbdc8c5-c678-4884-abe1-f02a76b83748 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:09.195666 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbdc8c5-c678-4884-abe1-f02a76b83748 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:09.195939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbdc8c5-c678-4884-abe1-f02a76b83748 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:09.207203 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbdc8c5-c678-4884-abe1-f02a76b83748 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:09.212234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-4cbdc8c5-c678-4884-abe1-f02a76b83748 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:27:09.212234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:09.212234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:09.212234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:09.212234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:09.212234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:09.212234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:09.212234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:09.212234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:09.212234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:09.212234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:09.212234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:09.212234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:09.212234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:09.212234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:09.212234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:09.213156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:09.213156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:09.213156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:09.213156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:09.213156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:09.213156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:09.213156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:09.213156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:09.213156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:09.213156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:09.213156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:09.213156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:09.213156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:09.213156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:09.213156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:09.213156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:09.214003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:09.214003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:09.214003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:09.214003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:09.214003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:09.214003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:09.214003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:09.214003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:09.214003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:09.214003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:09.214003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:27:09.214003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:09.214003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 166/341] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:09.219455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc959865-2a04-4341-b013-a1de36784cad None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:09.267724 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:09.334554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc959865-2a04-4341-b013-a1de36784cad None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:09.382015 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed67d4a7-af5d-4867-86ba-6f60dca34780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:09.382638 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed67d4a7-af5d-4867-86ba-6f60dca34780 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:09.382933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed67d4a7-af5d-4867-86ba-6f60dca34780 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:09.383216 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed67d4a7-af5d-4867-86ba-6f60dca34780 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:09.383887 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed67d4a7-af5d-4867-86ba-6f60dca34780 None admin] RBAC: Adding query filter params (name=member) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:09.384189 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed67d4a7-af5d-4867-86ba-6f60dca34780 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:09.385563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed67d4a7-af5d-4867-86ba-6f60dca34780 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:09.396880 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 176/342] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:27:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:09.402837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbdc8c5-c678-4884-abe1-f02a76b83748 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:09.493421 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:09.508906 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbdc8c5-c678-4884-abe1-f02a76b83748 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:09.535151 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed67d4a7-af5d-4867-86ba-6f60dca34780 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:09.554591 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184da8a0-b4d2-4fbc-9fba-f2b23cc5cc8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:09.555596 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184da8a0-b4d2-4fbc-9fba-f2b23cc5cc8d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:09.556068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184da8a0-b4d2-4fbc-9fba-f2b23cc5cc8d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:09.556517 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184da8a0-b4d2-4fbc-9fba-f2b23cc5cc8d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:09.682781 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed67d4a7-af5d-4867-86ba-6f60dca34780 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:09.692147 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:09.729985 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a8077d-505b-478a-a573-4e4273adf630 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:09.731061 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a8077d-505b-478a-a573-4e4273adf630 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:09.731339 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184da8a0-b4d2-4fbc-9fba-f2b23cc5cc8d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:09.731674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a8077d-505b-478a-a573-4e4273adf630 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:09.732096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a8077d-505b-478a-a573-4e4273adf630 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:09.733081 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184da8a0-b4d2-4fbc-9fba-f2b23cc5cc8d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:09.746264 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a8077d-505b-478a-a573-4e4273adf630 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:09.752141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-28a8077d-505b-478a-a573-4e4273adf630 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:27:09.752141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:09.752141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:09.752141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:09.752141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:09.752141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:09.752141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:09.752141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:09.752141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:09.752141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:09.752141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:09.752141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:09.752141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:09.752141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:09.752141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:09.752141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:09.753229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:09.753229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:09.753229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:09.753229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:09.753229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:09.753229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:09.753229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:09.753229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:09.753229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:09.753229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:09.753229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:09.753229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:09.753229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:09.753229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:09.753229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:09.753229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:09.754227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:09.754227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:09.754227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:09.754227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:09.754227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:09.754227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:09.754227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:09.754227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:09.754227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:09.754227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:09.754227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:27:09.754227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:09.756812 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 177/343] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:27:09.758290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a8077d-505b-478a-a573-4e4273adf630 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:09.896670 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a8077d-505b-478a-a573-4e4273adf630 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:09.898772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:09.917302 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 167/344] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 515 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:27:09.923007 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184da8a0-b4d2-4fbc-9fba-f2b23cc5cc8d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:09.950611 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8241a5b-1fa5-4d89-8713-77a715b690f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:09.951284 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8241a5b-1fa5-4d89-8713-77a715b690f2 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:09.951570 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8241a5b-1fa5-4d89-8713-77a715b690f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:09.951846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8241a5b-1fa5-4d89-8713-77a715b690f2 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:09.973009 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8241a5b-1fa5-4d89-8713-77a715b690f2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:09.978050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-d8241a5b-1fa5-4d89-8713-77a715b690f2 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:27:09.978050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:09.978050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:09.978050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:09.978050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:09.978050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:09.978050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:09.978050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:09.978050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:09.978050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:09.978050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:09.978050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:09.978050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:09.978050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:09.978050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:09.978050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:09.983693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:09.983693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:09.983693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:09.983693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:09.983693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:09.983693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:09.983693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:09.983693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:09.983693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:09.983693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:09.983693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:09.983693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:09.983693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:09.983693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:09.983693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:09.983693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:09.984532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:09.984532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:09.984532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:09.984532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:09.984532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:09.984532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:09.984532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:09.984532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:09.984532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:09.984532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:09.984532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:27:09.984532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:09.984532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 178/345] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:27:09 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:09.984532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8241a5b-1fa5-4d89-8713-77a715b690f2 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:10.034086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:10.039539 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184da8a0-b4d2-4fbc-9fba-f2b23cc5cc8d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:10.075827 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061091d1-efc4-4a74-b9c0-71f8888b0e04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:10.076890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061091d1-efc4-4a74-b9c0-71f8888b0e04 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:10.077329 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061091d1-efc4-4a74-b9c0-71f8888b0e04 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:10.077781 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061091d1-efc4-4a74-b9c0-71f8888b0e04 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:10.078654 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061091d1-efc4-4a74-b9c0-71f8888b0e04 None admin] RBAC: Adding query filter params (name=member) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:10.079763 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061091d1-efc4-4a74-b9c0-71f8888b0e04 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:10.081373 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061091d1-efc4-4a74-b9c0-71f8888b0e04 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:10.090732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 168/346] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:27:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:10.091898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061091d1-efc4-4a74-b9c0-71f8888b0e04 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:10.165931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8241a5b-1fa5-4d89-8713-77a715b690f2 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:10.195318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:10.241523 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5479950-4fbf-4053-bf87-1447790626fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:10.244160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5479950-4fbf-4053-bf87-1447790626fe None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:10.244624 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5479950-4fbf-4053-bf87-1447790626fe None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:10.245056 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5479950-4fbf-4053-bf87-1447790626fe None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:10.247782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5479950-4fbf-4053-bf87-1447790626fe None admin] RBAC: Authorizing `identity:get_project(project_id=8d00e05132b54e9b856788c06cdb7a56)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:10.249406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5479950-4fbf-4053-bf87-1447790626fe None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:10.254187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 179/347] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:27:09 2022] GET /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56 => generated 298 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:10.256112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5479950-4fbf-4053-bf87-1447790626fe None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:10.276791 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061091d1-efc4-4a74-b9c0-71f8888b0e04 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:10.317802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa575c4c-be5c-4d8e-854b-93bdce39b08c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:10.318755 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa575c4c-be5c-4d8e-854b-93bdce39b08c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:10.319235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa575c4c-be5c-4d8e-854b-93bdce39b08c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:10.319670 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa575c4c-be5c-4d8e-854b-93bdce39b08c None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:10.320547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa575c4c-be5c-4d8e-854b-93bdce39b08c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:10.321004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa575c4c-be5c-4d8e-854b-93bdce39b08c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:10.322613 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa575c4c-be5c-4d8e-854b-93bdce39b08c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:10.333553 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 169/348] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:10 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:10.334552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa575c4c-be5c-4d8e-854b-93bdce39b08c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:10.384057 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:10.390600 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5479950-4fbf-4053-bf87-1447790626fe None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:10.430760 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df9825b-c8b7-40ce-9dc0-0efe6fcde197 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:10.431424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df9825b-c8b7-40ce-9dc0-0efe6fcde197 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:10.431702 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df9825b-c8b7-40ce-9dc0-0efe6fcde197 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:10.432158 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df9825b-c8b7-40ce-9dc0-0efe6fcde197 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:10.451260 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa575c4c-be5c-4d8e-854b-93bdce39b08c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:10.467297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:10.489953 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2781dc-7ce1-4b60-93fe-074b0c0b66af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:10.491115 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2781dc-7ce1-4b60-93fe-074b0c0b66af None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:10.491611 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2781dc-7ce1-4b60-93fe-074b0c0b66af None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:10.492102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2781dc-7ce1-4b60-93fe-074b0c0b66af None admin] PATH_INFO: `/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:10.499033 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2781dc-7ce1-4b60-93fe-074b0c0b66af None admin] RBAC: Authorizing `identity:get_group(group_id=cd7cfa9c290c45a3bcd6bf544e905891)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:10.502778 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2781dc-7ce1-4b60-93fe-074b0c0b66af None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:10.504877 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 170/349] 10.222.0.49 () {56 vars in 1180 bytes} [Fri Sep 16 03:27:10 2022] GET /identity/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891 => generated 226 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:10.506574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ab3915-f44e-4f2f-b30d-318bf6efd616 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:10.507169 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ab3915-f44e-4f2f-b30d-318bf6efd616 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:10.507644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ab3915-f44e-4f2f-b30d-318bf6efd616 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:10.509326 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df9825b-c8b7-40ce-9dc0-0efe6fcde197 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:10.509653 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 171/350] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:10.510659 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ab3915-f44e-4f2f-b30d-318bf6efd616 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:10.510764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df9825b-c8b7-40ce-9dc0-0efe6fcde197 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:10.614067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:10.622858 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 180/351] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:27:10.624192 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f2c55f-22a6-44b6-99a3-7f9242007e05 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:10.624483 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f2c55f-22a6-44b6-99a3-7f9242007e05 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:10.624762 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f2c55f-22a6-44b6-99a3-7f9242007e05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:10.631148 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ab3915-f44e-4f2f-b30d-318bf6efd616 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:10.648358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-c3f2c55f-22a6-44b6-99a3-7f9242007e05 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:10.648358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c3f2c55f-22a6-44b6-99a3-7f9242007e05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:10.677600 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6500b482-4c5e-4445-b74d-563f2e20423c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:10.678695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6500b482-4c5e-4445-b74d-563f2e20423c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:10.679258 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6500b482-4c5e-4445-b74d-563f2e20423c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:10.679729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6500b482-4c5e-4445-b74d-563f2e20423c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:10.728761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 181/352] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:10.729622 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f2c55f-22a6-44b6-99a3-7f9242007e05 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:10.747355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:10.759308 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6500b482-4c5e-4445-b74d-563f2e20423c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:10.762311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6500b482-4c5e-4445-b74d-563f2e20423c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:10.840784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f2c55f-22a6-44b6-99a3-7f9242007e05 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:10.866516 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:10.878776 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e51c71-8e8c-433f-a974-acf7788d2312 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:10.879441 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e51c71-8e8c-433f-a974-acf7788d2312 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:10.887114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e51c71-8e8c-433f-a974-acf7788d2312 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:10.887114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e51c71-8e8c-433f-a974-acf7788d2312 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:10.887114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e51c71-8e8c-433f-a974-acf7788d2312 None admin] RBAC: Adding query filter params (group.id=cd7cfa9c290c45a3bcd6bf544e905891, role.id=589e1f1333a042b7a00c53384a2f5426, scope.project.id=8d00e05132b54e9b856788c06cdb7a56) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:10.887114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e51c71-8e8c-433f-a974-acf7788d2312 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:10.887114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e51c71-8e8c-433f-a974-acf7788d2312 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:10.889192 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 182/353] 10.222.0.49 () {56 vars in 1376 bytes} [Fri Sep 16 03:27:10 2022] GET /identity/v3/role_assignments?group.id=cd7cfa9c290c45a3bcd6bf544e905891&scope.project.id=8d00e05132b54e9b856788c06cdb7a56&role.id=589e1f1333a042b7a00c53384a2f5426 => generated 260 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:10.889724 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e51c71-8e8c-433f-a974-acf7788d2312 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:10.923061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 172/354] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:27:10.925328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6500b482-4c5e-4445-b74d-563f2e20423c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:11.030939 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e51c71-8e8c-433f-a974-acf7788d2312 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:11.055380 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:11.088745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6500b482-4c5e-4445-b74d-563f2e20423c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:11.088964 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e339564c-62ac-4ba8-b4bf-ed2feea7f33d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:11.089533 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e339564c-62ac-4ba8-b4bf-ed2feea7f33d None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:11.089828 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e339564c-62ac-4ba8-b4bf-ed2feea7f33d None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:11.090105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e339564c-62ac-4ba8-b4bf-ed2feea7f33d None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:11.095212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e339564c-62ac-4ba8-b4bf-ed2feea7f33d None admin] RBAC: Authorizing `identity:get_project(project_id=5f8ae3affd36481c8ce85ec30e7dedd9)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:11.095212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e339564c-62ac-4ba8-b4bf-ed2feea7f33d None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:11.095793 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 183/355] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:27:10 2022] GET /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9 => generated 302 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:11.101734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e339564c-62ac-4ba8-b4bf-ed2feea7f33d None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:11.154011 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af04809-38e8-48f9-acb5-ac540fc3c820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:11.155326 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af04809-38e8-48f9-acb5-ac540fc3c820 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:11.155831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af04809-38e8-48f9-acb5-ac540fc3c820 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:11.156344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af04809-38e8-48f9-acb5-ac540fc3c820 None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:11.158567 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af04809-38e8-48f9-acb5-ac540fc3c820 None admin] RBAC: Authorizing `identity:get_project(project_id=8d00e05132b54e9b856788c06cdb7a56)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:11.160302 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af04809-38e8-48f9-acb5-ac540fc3c820 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:11.166211 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 173/356] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:27:10 2022] GET /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56 => generated 298 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:11.168731 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af04809-38e8-48f9-acb5-ac540fc3c820 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:11.196238 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:11.256751 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e339564c-62ac-4ba8-b4bf-ed2feea7f33d None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:11.289907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af04809-38e8-48f9-acb5-ac540fc3c820 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:11.300356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898c15ec-e301-4f19-9c14-4e5e94e3d928 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:11.301374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898c15ec-e301-4f19-9c14-4e5e94e3d928 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:11.301667 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898c15ec-e301-4f19-9c14-4e5e94e3d928 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:11.301948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898c15ec-e301-4f19-9c14-4e5e94e3d928 None admin] PATH_INFO: `/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:11.303442 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898c15ec-e301-4f19-9c14-4e5e94e3d928 None admin] RBAC: Authorizing `identity:get_group(group_id=cd7cfa9c290c45a3bcd6bf544e905891)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:11.305585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898c15ec-e301-4f19-9c14-4e5e94e3d928 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:11.307378 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 184/357] 10.222.0.49 () {56 vars in 1180 bytes} [Fri Sep 16 03:27:11 2022] GET /identity/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891 => generated 226 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:11.308769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949d6534-5a40-4646-ba14-31fb4c1ea3bd None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:11.309753 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949d6534-5a40-4646-ba14-31fb4c1ea3bd None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:11.310041 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949d6534-5a40-4646-ba14-31fb4c1ea3bd None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:11.311978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 185/358] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:11.312233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949d6534-5a40-4646-ba14-31fb4c1ea3bd None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:11.342509 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cdbcfc5-8d24-4beb-94e3-937f660ab9cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:11.343192 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdbcfc5-8d24-4beb-94e3-937f660ab9cc None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:11.343470 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdbcfc5-8d24-4beb-94e3-937f660ab9cc None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:11.343738 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdbcfc5-8d24-4beb-94e3-937f660ab9cc None admin] PATH_INFO: `/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:11.344936 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cdbcfc5-8d24-4beb-94e3-937f660ab9cc None admin] RBAC: Authorizing `identity:get_group(group_id=cd7cfa9c290c45a3bcd6bf544e905891)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:11.346367 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cdbcfc5-8d24-4beb-94e3-937f660ab9cc None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:11.348268 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 174/359] 10.222.0.49 () {56 vars in 1180 bytes} [Fri Sep 16 03:27:11 2022] GET /identity/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891 => generated 226 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:11.349474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25730b96-3eab-40c1-8b11-355ddd2aca9b None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:11.349754 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25730b96-3eab-40c1-8b11-355ddd2aca9b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:11.350023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25730b96-3eab-40c1-8b11-355ddd2aca9b None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:11.351269 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 175/360] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:11 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:11.351928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25730b96-3eab-40c1-8b11-355ddd2aca9b None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:11.369074 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:11.444935 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949d6534-5a40-4646-ba14-31fb4c1ea3bd None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:11.472909 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:11.484299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25730b96-3eab-40c1-8b11-355ddd2aca9b None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:11.492522 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1375c7-3668-4eaf-ad70-30d8809a4408 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:11.493169 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1375c7-3668-4eaf-ad70-30d8809a4408 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:11.493446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1375c7-3668-4eaf-ad70-30d8809a4408 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:11.493711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1375c7-3668-4eaf-ad70-30d8809a4408 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:11.509052 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1375c7-3668-4eaf-ad70-30d8809a4408 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:11.514960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-2c1375c7-3668-4eaf-ad70-30d8809a4408 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:27:11.514960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:11.514960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:11.514960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:11.514960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:11.514960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:11.514960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:11.514960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:11.514960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:11.514960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:11.514960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:11.514960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:11.514960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:11.514960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:11.514960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:11.514960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:11.516043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:11.516043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:11.516043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:11.516043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:11.516043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:11.516043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:11.516043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:11.516043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:11.516043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:11.516043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:11.516043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:11.516043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:11.516043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:11.516043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:11.516043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:11.516043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:11.517620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:11.517620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:11.517620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:11.517620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:11.517620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:11.517620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:11.517620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:11.517620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:11.517620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:11.517620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:11.517620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:27:11.517620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:11.517620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 186/361] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:27:11 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:11.518296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4455c7-fde1-4f41-b047-bb190f0d2bbd None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:11.518375 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4455c7-fde1-4f41-b047-bb190f0d2bbd None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:11.518643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4455c7-fde1-4f41-b047-bb190f0d2bbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:11.532502 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8a386e-3ef5-4a27-b44a-01f75af46021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:11.533729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8a386e-3ef5-4a27-b44a-01f75af46021 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:11.534204 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8a386e-3ef5-4a27-b44a-01f75af46021 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:11.534666 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8a386e-3ef5-4a27-b44a-01f75af46021 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:11.535598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8a386e-3ef5-4a27-b44a-01f75af46021 None admin] RBAC: Adding query filter params (group.id=cd7cfa9c290c45a3bcd6bf544e905891, role.id=589e1f1333a042b7a00c53384a2f5426, scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:11.536129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8a386e-3ef5-4a27-b44a-01f75af46021 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:11.537760 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8a386e-3ef5-4a27-b44a-01f75af46021 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:11.542077 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-1e4455c7-fde1-4f41-b047-bb190f0d2bbd None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:11.543672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1e4455c7-fde1-4f41-b047-bb190f0d2bbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:11.548453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 176/362] 10.222.0.49 () {56 vars in 1376 bytes} [Fri Sep 16 03:27:11 2022] GET /identity/v3/role_assignments?group.id=cd7cfa9c290c45a3bcd6bf544e905891&scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9&role.id=589e1f1333a042b7a00c53384a2f5426 => generated 260 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:11.549532 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8475f0-c0b1-4a7d-94f5-f53cc446a55f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:11.549824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8475f0-c0b1-4a7d-94f5-f53cc446a55f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:11.550101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8475f0-c0b1-4a7d-94f5-f53cc446a55f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:11.573117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2b8475f0-c0b1-4a7d-94f5-f53cc446a55f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:11.574380 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2b8475f0-c0b1-4a7d-94f5-f53cc446a55f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:11.668261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 177/363] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:11.669248 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8475f0-c0b1-4a7d-94f5-f53cc446a55f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:11.676739 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 187/364] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:11.677700 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4455c7-fde1-4f41-b047-bb190f0d2bbd None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:11.678223 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:11.822432 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8475f0-c0b1-4a7d-94f5-f53cc446a55f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:11.891011 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db8be6f-35bd-45ed-941b-2de2112378ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:11.891935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db8be6f-35bd-45ed-941b-2de2112378ad None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:11.892385 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db8be6f-35bd-45ed-941b-2de2112378ad None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:11.892801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db8be6f-35bd-45ed-941b-2de2112378ad None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:11.893840 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db8be6f-35bd-45ed-941b-2de2112378ad None admin] RBAC: Adding query filter params (group.id=cd7cfa9c290c45a3bcd6bf544e905891, role.id=fa8bdf568182483aac8a14ef98a12a95, scope.project.id=8d00e05132b54e9b856788c06cdb7a56) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:11.895379 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db8be6f-35bd-45ed-941b-2de2112378ad None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:11.897651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db8be6f-35bd-45ed-941b-2de2112378ad None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:11.906940 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4455c7-fde1-4f41-b047-bb190f0d2bbd None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:11.913155 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 178/365] 10.222.0.49 () {56 vars in 1376 bytes} [Fri Sep 16 03:27:11 2022] GET /identity/v3/role_assignments?group.id=cd7cfa9c290c45a3bcd6bf544e905891&scope.project.id=8d00e05132b54e9b856788c06cdb7a56&role.id=fa8bdf568182483aac8a14ef98a12a95 => generated 260 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:11.924497 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:11.951818 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcfc15a-c6af-4fa6-a6cc-2dda9ece2fc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:11.952560 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcfc15a-c6af-4fa6-a6cc-2dda9ece2fc2 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:11.952851 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcfc15a-c6af-4fa6-a6cc-2dda9ece2fc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:11.953127 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcfc15a-c6af-4fa6-a6cc-2dda9ece2fc2 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:11.953811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcfc15a-c6af-4fa6-a6cc-2dda9ece2fc2 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:11.954232 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcfc15a-c6af-4fa6-a6cc-2dda9ece2fc2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:11.955785 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcfc15a-c6af-4fa6-a6cc-2dda9ece2fc2 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:11.968588 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 188/366] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:11 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:11.975026 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db8be6f-35bd-45ed-941b-2de2112378ad None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:12.031512 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcfc15a-c6af-4fa6-a6cc-2dda9ece2fc2 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:12.116479 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:12.190945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db8be6f-35bd-45ed-941b-2de2112378ad None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:12.248469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcfc15a-c6af-4fa6-a6cc-2dda9ece2fc2 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:12.261239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfb621c-6489-452c-8b7f-23fc5a54d777 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:12.262300 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfb621c-6489-452c-8b7f-23fc5a54d777 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:12.262779 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfb621c-6489-452c-8b7f-23fc5a54d777 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:12.263395 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfb621c-6489-452c-8b7f-23fc5a54d777 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:12.296579 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627920ab-1ebe-4ca0-8768-4f6e057ddb55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:12.297219 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627920ab-1ebe-4ca0-8768-4f6e057ddb55 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:12.297502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627920ab-1ebe-4ca0-8768-4f6e057ddb55 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:12.298420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627920ab-1ebe-4ca0-8768-4f6e057ddb55 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:12.334946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627920ab-1ebe-4ca0-8768-4f6e057ddb55 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:12.341218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-627920ab-1ebe-4ca0-8768-4f6e057ddb55 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:27:12.341218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:12.341218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:12.341218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:12.341218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:12.341218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:12.341218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:12.341218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:12.341218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:12.341218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:12.341218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:12.341218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:12.341218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:12.341218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:12.341218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:12.341218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:12.342329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:12.342329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:12.342329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:12.342329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:12.342329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:12.342329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:12.342329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:12.342329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:12.342329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:12.342329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:12.342329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:12.342329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:12.342329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:12.342329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:12.342329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:12.342329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:12.343175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:12.343175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:12.343175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:12.343175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:12.343175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:12.343175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:12.343175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:12.343175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:12.343175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:12.343175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:12.343175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:27:12.343175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:12.346170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 189/367] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:27:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:27:12.347175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627920ab-1ebe-4ca0-8768-4f6e057ddb55 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:12.418928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfb621c-6489-452c-8b7f-23fc5a54d777 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:12.420348 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfb621c-6489-452c-8b7f-23fc5a54d777 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:12.445562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:12.520955 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:12.582704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627920ab-1ebe-4ca0-8768-4f6e057ddb55 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:12.610058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 179/368] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 635 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:27:12.611838 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfb621c-6489-452c-8b7f-23fc5a54d777 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:12.644822 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393ae63d-4279-4656-81bf-654ae3b4b9fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:12.645499 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393ae63d-4279-4656-81bf-654ae3b4b9fc None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:12.645782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393ae63d-4279-4656-81bf-654ae3b4b9fc None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:12.646057 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393ae63d-4279-4656-81bf-654ae3b4b9fc None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:12.683752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-393ae63d-4279-4656-81bf-654ae3b4b9fc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:12.691403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-393ae63d-4279-4656-81bf-654ae3b4b9fc None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 03:27:12.691403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:12.691403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:12.691403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:12.691403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:12.691403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:12.691403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:12.691403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:12.691403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:12.691403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:12.691403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:12.691403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:12.691403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:12.691403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:12.691403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:12.691403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:12.692367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:12.692367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:12.692367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:12.692367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:12.692367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:12.692367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:12.692367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:12.692367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:12.692367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:12.692367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:12.692367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:12.692367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:12.692367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:12.692367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:12.692367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:12.692367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:12.693179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:12.693179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:12.693179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:12.693179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:12.693179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:12.693179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:12.693179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:12.693179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:12.693179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:12.693179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:12.693179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 03:27:12.693179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:12.695914 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 190/369] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:12 2022] GET /identity/v3/roles/reader => generated 84 bytes in 349 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:12.696956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393ae63d-4279-4656-81bf-654ae3b4b9fc None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:12.806939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfb621c-6489-452c-8b7f-23fc5a54d777 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:12.861690 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a42ba80-78e8-4697-a9da-354a523272a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:12.862715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a42ba80-78e8-4697-a9da-354a523272a5 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:12.863216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a42ba80-78e8-4697-a9da-354a523272a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:12.863596 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a42ba80-78e8-4697-a9da-354a523272a5 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:12.864292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a42ba80-78e8-4697-a9da-354a523272a5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:12.864944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a42ba80-78e8-4697-a9da-354a523272a5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:12.867520 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a42ba80-78e8-4697-a9da-354a523272a5 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:12.897996 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 180/370] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:12.898692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a42ba80-78e8-4697-a9da-354a523272a5 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:12.947085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:12.952198 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393ae63d-4279-4656-81bf-654ae3b4b9fc None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:12.988280 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:13.014841 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2b1ea9-1546-46e1-bb0c-126fd03d0878 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:13.016127 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2b1ea9-1546-46e1-bb0c-126fd03d0878 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:13.017020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2b1ea9-1546-46e1-bb0c-126fd03d0878 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:13.017317 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2b1ea9-1546-46e1-bb0c-126fd03d0878 None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:13.018583 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2b1ea9-1546-46e1-bb0c-126fd03d0878 None admin] RBAC: Authorizing `identity:get_project(project_id=5f8ae3affd36481c8ce85ec30e7dedd9)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:13.024053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2b1ea9-1546-46e1-bb0c-126fd03d0878 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:13.026211 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 191/371] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:27:12 2022] GET /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9 => generated 302 bytes in 330 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:13.027507 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2b1ea9-1546-46e1-bb0c-126fd03d0878 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:13.194811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2b1ea9-1546-46e1-bb0c-126fd03d0878 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:13.196098 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a42ba80-78e8-4697-a9da-354a523272a5 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:13.246278 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e90347-dd67-489c-a049-d445b75bdefb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:13.247182 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e90347-dd67-489c-a049-d445b75bdefb None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:13.247478 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e90347-dd67-489c-a049-d445b75bdefb None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:13.247762 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e90347-dd67-489c-a049-d445b75bdefb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:13.325688 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f716b2b0-ed47-46e7-b87b-3cd0864f0a20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:13.326326 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f716b2b0-ed47-46e7-b87b-3cd0864f0a20 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:13.326605 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f716b2b0-ed47-46e7-b87b-3cd0864f0a20 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:13.326871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f716b2b0-ed47-46e7-b87b-3cd0864f0a20 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:13.327563 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f716b2b0-ed47-46e7-b87b-3cd0864f0a20 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:13.327860 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f716b2b0-ed47-46e7-b87b-3cd0864f0a20 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:13.329461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f716b2b0-ed47-46e7-b87b-3cd0864f0a20 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:13.329896 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:13.353375 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 181/372] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:27:12 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 455 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:13.354324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f716b2b0-ed47-46e7-b87b-3cd0864f0a20 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:13.354750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e90347-dd67-489c-a049-d445b75bdefb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:13.356599 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e90347-dd67-489c-a049-d445b75bdefb None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:13.515086 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:13.524756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f716b2b0-ed47-46e7-b87b-3cd0864f0a20 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:13.552933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 192/373] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 526 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:27:13.553799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e90347-dd67-489c-a049-d445b75bdefb None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:13.566872 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53093e1-c89f-41ca-ae0f-700aa1c78a00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:13.567605 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53093e1-c89f-41ca-ae0f-700aa1c78a00 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:13.568359 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53093e1-c89f-41ca-ae0f-700aa1c78a00 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:13.568665 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53093e1-c89f-41ca-ae0f-700aa1c78a00 None admin] PATH_INFO: `/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:13.569829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53093e1-c89f-41ca-ae0f-700aa1c78a00 None admin] RBAC: Authorizing `identity:get_group(group_id=cd7cfa9c290c45a3bcd6bf544e905891)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:13.571310 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53093e1-c89f-41ca-ae0f-700aa1c78a00 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:13.573636 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 182/374] 10.222.0.49 () {56 vars in 1180 bytes} [Fri Sep 16 03:27:13 2022] GET /identity/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891 => generated 226 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:13.574470 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53093e1-c89f-41ca-ae0f-700aa1c78a00 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:13.694361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53093e1-c89f-41ca-ae0f-700aa1c78a00 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:13.734184 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2b067a-498a-443e-81e6-711f3b0ee39f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:13.734962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2b067a-498a-443e-81e6-711f3b0ee39f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:13.735840 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2b067a-498a-443e-81e6-711f3b0ee39f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:13.736239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2b067a-498a-443e-81e6-711f3b0ee39f None admin] PATH_INFO: `/v3/projects/ccdd7a44ba5049fb939dad44d5bd406e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:13.736531 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:13.737687 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2b067a-498a-443e-81e6-711f3b0ee39f None admin] RBAC: Authorizing `identity:get_project(project_id=ccdd7a44ba5049fb939dad44d5bd406e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:13.739240 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2b067a-498a-443e-81e6-711f3b0ee39f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:13.741042 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 183/375] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:27:13 2022] GET /identity/v3/projects/ccdd7a44ba5049fb939dad44d5bd406e => generated 344 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:13.741891 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2b067a-498a-443e-81e6-711f3b0ee39f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:13.779169 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:13.806093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e90347-dd67-489c-a049-d445b75bdefb None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:13.849649 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987eef90-87d7-4b96-ab22-71cbbd3cab0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:13.850296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987eef90-87d7-4b96-ab22-71cbbd3cab0a None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:13.850579 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987eef90-87d7-4b96-ab22-71cbbd3cab0a None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:13.850852 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987eef90-87d7-4b96-ab22-71cbbd3cab0a None admin] PATH_INFO: `/v3/users/3af08fdc88fd48fe91b3e2370bbe8745` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:13.852074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987eef90-87d7-4b96-ab22-71cbbd3cab0a None admin] RBAC: Authorizing `identity:get_user(user_id=3af08fdc88fd48fe91b3e2370bbe8745)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:13.853524 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987eef90-87d7-4b96-ab22-71cbbd3cab0a None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:13.855429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 193/376] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:27:13 2022] GET /identity/v3/users/3af08fdc88fd48fe91b3e2370bbe8745 => generated 293 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:13.863156 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987eef90-87d7-4b96-ab22-71cbbd3cab0a None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:13.872227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2b067a-498a-443e-81e6-711f3b0ee39f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:13.917766 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b9645e-15a9-48f5-9e17-850d1ea2d487 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:13.918372 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b9645e-15a9-48f5-9e17-850d1ea2d487 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:13.918690 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b9645e-15a9-48f5-9e17-850d1ea2d487 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:13.918993 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b9645e-15a9-48f5-9e17-850d1ea2d487 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:13.919723 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b9645e-15a9-48f5-9e17-850d1ea2d487 None admin] RBAC: Adding query filter params (group.id=cd7cfa9c290c45a3bcd6bf544e905891, role.id=fa8bdf568182483aac8a14ef98a12a95, scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:13.920086 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b9645e-15a9-48f5-9e17-850d1ea2d487 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:13.921513 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b9645e-15a9-48f5-9e17-850d1ea2d487 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:13.934349 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 184/377] 10.222.0.49 () {56 vars in 1376 bytes} [Fri Sep 16 03:27:13 2022] GET /identity/v3/role_assignments?group.id=cd7cfa9c290c45a3bcd6bf544e905891&scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9&role.id=fa8bdf568182483aac8a14ef98a12a95 => generated 260 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:13.935408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b9645e-15a9-48f5-9e17-850d1ea2d487 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:13.974611 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987eef90-87d7-4b96-ab22-71cbbd3cab0a None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:14.009419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:14.020441 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f05c20d-2ebb-43eb-bd00-dfb873900a12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:14.022462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f05c20d-2ebb-43eb-bd00-dfb873900a12 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:14.022462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f05c20d-2ebb-43eb-bd00-dfb873900a12 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:14.022462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f05c20d-2ebb-43eb-bd00-dfb873900a12 None admin] PATH_INFO: `/v3/groups/88b3e22965ff4c70a94e3b19fa5b6ed6` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:14.037018 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f05c20d-2ebb-43eb-bd00-dfb873900a12 None admin] RBAC: Authorizing `identity:get_group(group_id=88b3e22965ff4c70a94e3b19fa5b6ed6)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:14.038776 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f05c20d-2ebb-43eb-bd00-dfb873900a12 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:14.042432 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 194/378] 10.222.0.49 () {56 vars in 1180 bytes} [Fri Sep 16 03:27:13 2022] GET /identity/v3/groups/88b3e22965ff4c70a94e3b19fa5b6ed6 => generated 229 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:14.043739 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9da59e-6269-421b-a6b9-12dbfa8ddb9e None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:14.044091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9da59e-6269-421b-a6b9-12dbfa8ddb9e None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:14.044367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9da59e-6269-421b-a6b9-12dbfa8ddb9e None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:14.045939 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 195/379] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:14.047732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9da59e-6269-421b-a6b9-12dbfa8ddb9e None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:14.072015 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:14.081952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b9645e-15a9-48f5-9e17-850d1ea2d487 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:14.127808 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fcb963-a5e5-4525-9f4e-ed1afea2ade4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:14.128438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fcb963-a5e5-4525-9f4e-ed1afea2ade4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:14.128719 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fcb963-a5e5-4525-9f4e-ed1afea2ade4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:14.128997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fcb963-a5e5-4525-9f4e-ed1afea2ade4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:14.129903 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94fcb963-a5e5-4525-9f4e-ed1afea2ade4 None admin] RBAC: Adding query filter params (role.id=363c2654ab4e456b8b9b0b8c8c28911e, user.id=3af08fdc88fd48fe91b3e2370bbe8745, scope.system=all) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:14.130110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94fcb963-a5e5-4525-9f4e-ed1afea2ade4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:14.132636 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94fcb963-a5e5-4525-9f4e-ed1afea2ade4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:14.150345 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 185/380] 10.222.0.49 () {56 vars in 1308 bytes} [Fri Sep 16 03:27:13 2022] GET /identity/v3/role_assignments?user.id=3af08fdc88fd48fe91b3e2370bbe8745&scope.system=all&role.id=363c2654ab4e456b8b9b0b8c8c28911e => generated 510 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:14.152625 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabe2990-7183-4356-9f09-037b942310f2 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:14.152976 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabe2990-7183-4356-9f09-037b942310f2 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:14.153258 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabe2990-7183-4356-9f09-037b942310f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:14.195629 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-cabe2990-7183-4356-9f09-037b942310f2 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:14.201163 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cabe2990-7183-4356-9f09-037b942310f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:14.221092 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9da59e-6269-421b-a6b9-12dbfa8ddb9e None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:14.283933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27be40a1-7869-42f3-9027-4312854e447f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:14.284565 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27be40a1-7869-42f3-9027-4312854e447f None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:14.284848 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27be40a1-7869-42f3-9027-4312854e447f None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:14.285129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27be40a1-7869-42f3-9027-4312854e447f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:14.286036 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27be40a1-7869-42f3-9027-4312854e447f None admin] RBAC: Adding query filter params (group.id=88b3e22965ff4c70a94e3b19fa5b6ed6, role.id=89c4b3a971224d468d33d2dc411230a2, scope.project.id=ccdd7a44ba5049fb939dad44d5bd406e) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:14.286165 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27be40a1-7869-42f3-9027-4312854e447f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:14.287767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27be40a1-7869-42f3-9027-4312854e447f None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:14.298335 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 196/381] 10.222.0.49 () {56 vars in 1376 bytes} [Fri Sep 16 03:27:14 2022] GET /identity/v3/role_assignments?group.id=88b3e22965ff4c70a94e3b19fa5b6ed6&scope.project.id=ccdd7a44ba5049fb939dad44d5bd406e&role.id=89c4b3a971224d468d33d2dc411230a2 => generated 260 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:14.309287 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 186/382] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:14.677868 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27be40a1-7869-42f3-9027-4312854e447f None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:14.746033 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd574657-afc4-4423-b1fc-76b733b9e830 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:14.747001 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd574657-afc4-4423-b1fc-76b733b9e830 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:14.747473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd574657-afc4-4423-b1fc-76b733b9e830 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:14.749018 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 187/383] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:27:14.756479 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcfa651-33ee-4737-8319-589a1e96679e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:14.757156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcfa651-33ee-4737-8319-589a1e96679e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:14.757625 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcfa651-33ee-4737-8319-589a1e96679e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:14.786082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:14.794683 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-afcfa651-33ee-4737-8319-589a1e96679e None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:14.796149 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-afcfa651-33ee-4737-8319-589a1e96679e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:14.866923 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27be40a1-7869-42f3-9027-4312854e447f None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:14.878550 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 188/384] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Sep 16 03:27:14.928279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0174bcd6-d2cb-4e4c-a752-9506f4e8f3a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:14.928845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0174bcd6-d2cb-4e4c-a752-9506f4e8f3a1 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:14.929128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0174bcd6-d2cb-4e4c-a752-9506f4e8f3a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:14.929399 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0174bcd6-d2cb-4e4c-a752-9506f4e8f3a1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:14.947771 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0174bcd6-d2cb-4e4c-a752-9506f4e8f3a1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:14.953034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-0174bcd6-d2cb-4e4c-a752-9506f4e8f3a1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:27:14.953034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:14.953034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:14.953034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:14.953034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:14.953034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:14.953034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:14.953034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:14.953034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:14.953034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:14.953034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:14.953034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:14.953034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:14.953034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:14.953034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:14.953034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:14.954033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:14.954033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:14.954033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:14.954033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:14.954033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:14.954033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:14.954033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:14.954033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:14.954033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:14.954033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:14.954033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:14.954033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:14.954033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:14.954033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:14.954033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:14.954033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:14.955520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:14.955520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:14.955520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:14.955520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:14.955520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:14.955520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:14.955520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:14.955520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:14.955520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:14.955520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:14.955520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:27:14.955520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:14.955520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 197/385] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:14 2022] GET /identity/v3/roles/member => generated 84 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:14.960058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcfa651-33ee-4737-8319-589a1e96679e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:14.979281 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:15.104558 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcfa651-33ee-4737-8319-589a1e96679e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:15.144770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb66959-82ed-4859-a146-8952c54d6ba1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:15.145753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb66959-82ed-4859-a146-8952c54d6ba1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:15.146269 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb66959-82ed-4859-a146-8952c54d6ba1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:15.146734 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb66959-82ed-4859-a146-8952c54d6ba1 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:15.147610 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb66959-82ed-4859-a146-8952c54d6ba1 None admin] RBAC: Adding query filter params (name=member) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:15.148125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb66959-82ed-4859-a146-8952c54d6ba1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:15.149688 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb66959-82ed-4859-a146-8952c54d6ba1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:15.162420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 189/386] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:27:14 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:15.165604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0174bcd6-d2cb-4e4c-a752-9506f4e8f3a1 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:15.319409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb66959-82ed-4859-a146-8952c54d6ba1 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:15.334121 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0174bcd6-d2cb-4e4c-a752-9506f4e8f3a1 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:15.379617 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:15.404001 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:15.419617 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9abfb80-599c-42da-9fa1-ab7ca7027189 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:15.420573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9abfb80-599c-42da-9fa1-ab7ca7027189 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:15.420854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9abfb80-599c-42da-9fa1-ab7ca7027189 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:15.421130 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9abfb80-599c-42da-9fa1-ab7ca7027189 None admin] PATH_INFO: `/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:15.423800 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9abfb80-599c-42da-9fa1-ab7ca7027189 None admin] RBAC: Authorizing `identity:get_group(group_id=cd7cfa9c290c45a3bcd6bf544e905891)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:15.426234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9abfb80-599c-42da-9fa1-ab7ca7027189 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:15.431304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 198/387] 10.222.0.49 () {56 vars in 1180 bytes} [Fri Sep 16 03:27:15 2022] GET /identity/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891 => generated 226 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:15.432917 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1e7410-70e1-4656-9861-55d47cb12f4e None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:15.433710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1e7410-70e1-4656-9861-55d47cb12f4e None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:15.434158 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1e7410-70e1-4656-9861-55d47cb12f4e None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:15.437179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 199/388] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:15.441212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1e7410-70e1-4656-9861-55d47cb12f4e None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:15.448596 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb66959-82ed-4859-a146-8952c54d6ba1 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:15.494472 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e26c711-b5c3-4a87-8337-09343be8c36c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:15.495417 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e26c711-b5c3-4a87-8337-09343be8c36c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:15.495968 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e26c711-b5c3-4a87-8337-09343be8c36c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:15.496455 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e26c711-b5c3-4a87-8337-09343be8c36c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:15.508514 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e26c711-b5c3-4a87-8337-09343be8c36c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:15.514505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-0e26c711-b5c3-4a87-8337-09343be8c36c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:27:15.514505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:15.514505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:15.514505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:15.514505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:15.514505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:15.514505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:15.514505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:15.514505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:15.514505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:15.514505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:15.514505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:15.514505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:15.514505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:15.514505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:15.514505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:15.515438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:15.515438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:15.515438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:15.515438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:15.515438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:15.515438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:15.515438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:15.515438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:15.515438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:15.515438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:15.515438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:15.515438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:15.515438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:15.515438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:15.515438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:15.515438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:15.516252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:15.516252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:15.516252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:15.516252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:15.516252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:15.516252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:15.516252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:15.516252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:15.516252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:15.516252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:15.516252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:27:15.516252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:15.519066 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 190/389] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:15.520652 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78335d4-9930-4605-a303-6e0a6ba05132 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:15.521153 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78335d4-9930-4605-a303-6e0a6ba05132 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:15.521713 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78335d4-9930-4605-a303-6e0a6ba05132 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:15.547605 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-c78335d4-9930-4605-a303-6e0a6ba05132 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:15.551015 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c78335d4-9930-4605-a303-6e0a6ba05132 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:15.577594 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1e7410-70e1-4656-9861-55d47cb12f4e None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:15.615914 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14224633-3272-4d24-86a3-e49bc6dcb46d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:15.616876 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14224633-3272-4d24-86a3-e49bc6dcb46d None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:15.619433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14224633-3272-4d24-86a3-e49bc6dcb46d None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:15.619953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14224633-3272-4d24-86a3-e49bc6dcb46d None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:15.621404 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14224633-3272-4d24-86a3-e49bc6dcb46d None admin] RBAC: Authorizing `identity:get_project(project_id=8d00e05132b54e9b856788c06cdb7a56)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:15.623073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14224633-3272-4d24-86a3-e49bc6dcb46d None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:15.626539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 200/390] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:27:15 2022] GET /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56 => generated 298 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:15.627712 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14224633-3272-4d24-86a3-e49bc6dcb46d None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:15.635170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 191/391] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:15.636056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78335d4-9930-4605-a303-6e0a6ba05132 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:15.731601 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:15.748990 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:15.810500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14224633-3272-4d24-86a3-e49bc6dcb46d None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:15.826586 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78335d4-9930-4605-a303-6e0a6ba05132 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:15.885411 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73589ae4-f814-4834-b002-979addb32f72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:15.885943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73589ae4-f814-4834-b002-979addb32f72 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:15.886221 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73589ae4-f814-4834-b002-979addb32f72 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:15.886494 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73589ae4-f814-4834-b002-979addb32f72 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:15.887191 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73589ae4-f814-4834-b002-979addb32f72 None admin] RBAC: Adding query filter params (name=member) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:15.887493 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73589ae4-f814-4834-b002-979addb32f72 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:15.888881 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73589ae4-f814-4834-b002-979addb32f72 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:15.898775 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcbbbf3-4db6-4245-8841-9641df98c02e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:15.899427 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcbbbf3-4db6-4245-8841-9641df98c02e None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:15.899707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcbbbf3-4db6-4245-8841-9641df98c02e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:15.899978 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcbbbf3-4db6-4245-8841-9641df98c02e None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56/groups/cd7cfa9c290c45a3bcd6bf544e905891/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:15.902859 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcbbbf3-4db6-4245-8841-9641df98c02e None admin] RBAC: Authorizing `identity:create_grant(project_id=8d00e05132b54e9b856788c06cdb7a56, group_id=cd7cfa9c290c45a3bcd6bf544e905891, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:15.904625 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcbbbf3-4db6-4245-8841-9641df98c02e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:15.909408 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 201/392] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:27:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:15.915897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73589ae4-f814-4834-b002-979addb32f72 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:15.977444 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 192/393] 10.222.0.49 () {58 vars in 1442 bytes} [Fri Sep 16 03:27:15 2022] PUT /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56/groups/cd7cfa9c290c45a3bcd6bf544e905891/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:27:16.017011 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcbbbf3-4db6-4245-8841-9641df98c02e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:16.044469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73589ae4-f814-4834-b002-979addb32f72 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:16.065267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:16.088205 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b077315-eb32-4721-82ff-0da76dd3772f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:16.090729 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b077315-eb32-4721-82ff-0da76dd3772f None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:16.091694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b077315-eb32-4721-82ff-0da76dd3772f None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:16.092154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b077315-eb32-4721-82ff-0da76dd3772f None admin] PATH_INFO: `/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:16.094569 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b077315-eb32-4721-82ff-0da76dd3772f None admin] RBAC: Authorizing `identity:get_group(group_id=cd7cfa9c290c45a3bcd6bf544e905891)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:16.097098 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b077315-eb32-4721-82ff-0da76dd3772f None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:16.101921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 202/394] 10.222.0.49 () {56 vars in 1180 bytes} [Fri Sep 16 03:27:15 2022] GET /identity/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891 => generated 226 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:16.112670 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b077315-eb32-4721-82ff-0da76dd3772f None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:16.201460 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcbbbf3-4db6-4245-8841-9641df98c02e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:16.227172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:16.240668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0d2ce3-4a07-455d-9c34-5010fb936932 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:16.242441 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0d2ce3-4a07-455d-9c34-5010fb936932 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:16.242441 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0d2ce3-4a07-455d-9c34-5010fb936932 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:16.242441 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0d2ce3-4a07-455d-9c34-5010fb936932 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:16.254838 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0d2ce3-4a07-455d-9c34-5010fb936932 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:16.260776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-2a0d2ce3-4a07-455d-9c34-5010fb936932 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:27:16.260776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:16.260776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:16.260776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:16.260776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:16.260776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:16.260776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:16.260776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:16.260776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:16.260776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:16.260776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:16.260776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:16.260776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:16.260776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:16.260776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:16.260776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:16.261714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:16.261714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:16.261714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:16.261714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:16.261714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:16.261714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:16.261714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:16.261714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:16.261714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:16.261714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:16.261714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:16.261714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:16.261714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:16.261714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:16.261714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:16.261714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:16.262598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:16.262598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:16.262598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:16.262598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:16.262598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:16.262598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:16.262598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:16.262598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:16.262598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:16.262598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:16.262598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:27:16.262598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:16.262598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 193/395] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:27:16 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:27:16.267959 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0d2ce3-4a07-455d-9c34-5010fb936932 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:16.281389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b077315-eb32-4721-82ff-0da76dd3772f None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:16.341380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76bac97-df0f-4a5a-ab15-8f0441e4d0f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:16.342323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76bac97-df0f-4a5a-ab15-8f0441e4d0f5 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:16.342609 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76bac97-df0f-4a5a-ab15-8f0441e4d0f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:16.342890 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76bac97-df0f-4a5a-ab15-8f0441e4d0f5 None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:16.344493 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76bac97-df0f-4a5a-ab15-8f0441e4d0f5 None admin] RBAC: Authorizing `identity:get_project(project_id=5f8ae3affd36481c8ce85ec30e7dedd9)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:16.347198 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76bac97-df0f-4a5a-ab15-8f0441e4d0f5 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:16.349592 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 203/396] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:27:16 2022] GET /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9 => generated 302 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:16.355355 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76bac97-df0f-4a5a-ab15-8f0441e4d0f5 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:16.397796 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0d2ce3-4a07-455d-9c34-5010fb936932 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:16.420711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:16.435579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26eb95b-43d9-4de4-83f8-b18831609fea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:16.436299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26eb95b-43d9-4de4-83f8-b18831609fea None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:16.436673 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26eb95b-43d9-4de4-83f8-b18831609fea None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:16.437081 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26eb95b-43d9-4de4-83f8-b18831609fea None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:16.437730 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26eb95b-43d9-4de4-83f8-b18831609fea None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:16.439410 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26eb95b-43d9-4de4-83f8-b18831609fea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:16.440979 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26eb95b-43d9-4de4-83f8-b18831609fea None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:16.449953 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 194/397] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:16 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:16.455945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26eb95b-43d9-4de4-83f8-b18831609fea None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:16.474540 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76bac97-df0f-4a5a-ab15-8f0441e4d0f5 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:16.511263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:16.512514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15fac55-b51e-4613-8b7d-f2b545bba5cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:16.513237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15fac55-b51e-4613-8b7d-f2b545bba5cd None admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:16.513522 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15fac55-b51e-4613-8b7d-f2b545bba5cd None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:16.513800 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15fac55-b51e-4613-8b7d-f2b545bba5cd None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9/groups/cd7cfa9c290c45a3bcd6bf544e905891/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:16.516487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15fac55-b51e-4613-8b7d-f2b545bba5cd None admin] RBAC: Authorizing `identity:create_grant(project_id=5f8ae3affd36481c8ce85ec30e7dedd9, group_id=cd7cfa9c290c45a3bcd6bf544e905891, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:16.518114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15fac55-b51e-4613-8b7d-f2b545bba5cd None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:16.569046 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26eb95b-43d9-4de4-83f8-b18831609fea None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:16.604845 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237a43b5-7508-4d2c-84f9-3c798354c00e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:16.605485 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237a43b5-7508-4d2c-84f9-3c798354c00e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:16.605762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237a43b5-7508-4d2c-84f9-3c798354c00e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:16.606036 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237a43b5-7508-4d2c-84f9-3c798354c00e None admin] PATH_INFO: `/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:16.609011 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237a43b5-7508-4d2c-84f9-3c798354c00e None admin] RBAC: Authorizing `identity:get_group(group_id=cd7cfa9c290c45a3bcd6bf544e905891)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:16.611218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237a43b5-7508-4d2c-84f9-3c798354c00e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:16.613629 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 195/398] 10.222.0.49 () {56 vars in 1180 bytes} [Fri Sep 16 03:27:16 2022] GET /identity/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891 => generated 226 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:16.618829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237a43b5-7508-4d2c-84f9-3c798354c00e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:16.621495 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 204/399] 10.222.0.49 () {58 vars in 1442 bytes} [Fri Sep 16 03:27:16 2022] PUT /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9/groups/cd7cfa9c290c45a3bcd6bf544e905891/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:27:16.733629 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237a43b5-7508-4d2c-84f9-3c798354c00e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:16.749585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:16.772112 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9999550d-ed8f-44ce-a85e-0f189b82f268 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:16.772766 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9999550d-ed8f-44ce-a85e-0f189b82f268 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:16.773048 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9999550d-ed8f-44ce-a85e-0f189b82f268 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:16.773322 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9999550d-ed8f-44ce-a85e-0f189b82f268 None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:16.774514 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9999550d-ed8f-44ce-a85e-0f189b82f268 None admin] RBAC: Authorizing `identity:get_project(project_id=8d00e05132b54e9b856788c06cdb7a56)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:16.776061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9999550d-ed8f-44ce-a85e-0f189b82f268 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:16.778087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 196/400] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:27:16 2022] GET /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56 => generated 298 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:16.784168 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15fac55-b51e-4613-8b7d-f2b545bba5cd None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:16.930388 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15fac55-b51e-4613-8b7d-f2b545bba5cd None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:16.947512 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:16.972902 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ece8c33-3adb-44ef-81f5-d83289ed1252 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:16.974007 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ece8c33-3adb-44ef-81f5-d83289ed1252 None admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:16.974555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ece8c33-3adb-44ef-81f5-d83289ed1252 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:16.975156 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ece8c33-3adb-44ef-81f5-d83289ed1252 None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56/groups/cd7cfa9c290c45a3bcd6bf544e905891/roles/fa8bdf568182483aac8a14ef98a12a95` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:16.977966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ece8c33-3adb-44ef-81f5-d83289ed1252 None admin] RBAC: Authorizing `identity:create_grant(project_id=8d00e05132b54e9b856788c06cdb7a56, group_id=cd7cfa9c290c45a3bcd6bf544e905891, role_id=fa8bdf568182483aac8a14ef98a12a95)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:16.979959 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ece8c33-3adb-44ef-81f5-d83289ed1252 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:17.083756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c162bf20-bf54-4c68-b19a-fdcb43f149d6 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:17.084368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c162bf20-bf54-4c68-b19a-fdcb43f149d6 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:17.084846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c162bf20-bf54-4c68-b19a-fdcb43f149d6 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:17.086602 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 197/401] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:17.092277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e5013c-b8cb-427a-bf7c-e9ba014d5854 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:17.092567 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e5013c-b8cb-427a-bf7c-e9ba014d5854 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:17.092849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e5013c-b8cb-427a-bf7c-e9ba014d5854 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:17.112942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 205/402] 10.222.0.49 () {58 vars in 1442 bytes} [Fri Sep 16 03:27:16 2022] PUT /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56/groups/cd7cfa9c290c45a3bcd6bf544e905891/roles/fa8bdf568182483aac8a14ef98a12a95 => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:27:17.118845 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-12e5013c-b8cb-427a-bf7c-e9ba014d5854 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:17.120111 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-12e5013c-b8cb-427a-bf7c-e9ba014d5854 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:17.223854 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 198/403] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 03:27:17.224193 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1c3886-9b82-4864-8c04-11a3572b5acc None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:17.224439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1c3886-9b82-4864-8c04-11a3572b5acc None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:17.224717 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1c3886-9b82-4864-8c04-11a3572b5acc None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:17.226145 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 206/404] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:27:17.232440 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b9d369-049d-42da-88e4-2698610116e7 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:17.232761 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b9d369-049d-42da-88e4-2698610116e7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:17.232988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b9d369-049d-42da-88e4-2698610116e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:17.257580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e6b9d369-049d-42da-88e4-2698610116e7 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:17.258786 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e6b9d369-049d-42da-88e4-2698610116e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:17.357374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 199/405] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:17.536295 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1c3886-9b82-4864-8c04-11a3572b5acc None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:17.664263 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1c3886-9b82-4864-8c04-11a3572b5acc None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:17.702080 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b9d369-049d-42da-88e4-2698610116e7 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:17.704946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fdbd3b-142b-440d-a069-d6f4dd0de0b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:17.705593 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fdbd3b-142b-440d-a069-d6f4dd0de0b2 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:17.705878 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fdbd3b-142b-440d-a069-d6f4dd0de0b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:17.706151 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fdbd3b-142b-440d-a069-d6f4dd0de0b2 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:17.723621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fdbd3b-142b-440d-a069-d6f4dd0de0b2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:17.730285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-c4fdbd3b-142b-440d-a069-d6f4dd0de0b2 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:27:17.730285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:17.730285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:17.730285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:17.730285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:17.730285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:17.730285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:17.730285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:17.730285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:17.730285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:17.730285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:17.730285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:17.730285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:17.730285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:17.730285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:17.730285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:17.731234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:17.731234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:17.731234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:17.731234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:17.731234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:17.731234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:17.731234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:17.731234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:17.731234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:17.731234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:17.731234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:17.731234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:17.731234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:17.731234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:17.731234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:17.731234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:17.732046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:17.732046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:17.732046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:17.732046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:17.732046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:17.732046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:17.732046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:17.732046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:17.732046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:17.732046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:17.732046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:27:17.732046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:17.732695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 207/406] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:27:17 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:17.740950 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fdbd3b-142b-440d-a069-d6f4dd0de0b2 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:17.780049 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:17.808163 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:17.864853 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b9d369-049d-42da-88e4-2698610116e7 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:17.926379 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c54d1da-aa6f-4968-a4a6-062ae562b48d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:17.926965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c54d1da-aa6f-4968-a4a6-062ae562b48d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:17.927225 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c54d1da-aa6f-4968-a4a6-062ae562b48d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:17.927495 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c54d1da-aa6f-4968-a4a6-062ae562b48d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:17.929978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fdbd3b-142b-440d-a069-d6f4dd0de0b2 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:17.945966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c54d1da-aa6f-4968-a4a6-062ae562b48d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:17.950776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-2c54d1da-aa6f-4968-a4a6-062ae562b48d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:27:17.950776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:17.950776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:17.950776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:17.950776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:17.950776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:17.950776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:17.950776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:17.950776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:17.950776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:17.950776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:17.950776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:17.950776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:17.950776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:17.950776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:17.950776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:17.951901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:17.951901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:17.951901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:17.951901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:17.951901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:17.951901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:17.951901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:17.951901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:17.951901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:17.951901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:17.951901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:17.951901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:17.951901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:17.951901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:17.951901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:17.951901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:17.952926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:17.952926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:17.952926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:17.952926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:17.952926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:17.952926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:17.952926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:17.952926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:17.952926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:17.952926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:17.952926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:27:17.952926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:17.960320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 200/407] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:27:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:17.965839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c54d1da-aa6f-4968-a4a6-062ae562b48d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:17.973633 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565bcd9b-a705-4630-ab2e-5b6a8d6410c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:17.974809 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565bcd9b-a705-4630-ab2e-5b6a8d6410c9 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:17.975279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565bcd9b-a705-4630-ab2e-5b6a8d6410c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:17.975750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565bcd9b-a705-4630-ab2e-5b6a8d6410c9 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:17.976557 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565bcd9b-a705-4630-ab2e-5b6a8d6410c9 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:17.977029 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565bcd9b-a705-4630-ab2e-5b6a8d6410c9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:17.978670 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565bcd9b-a705-4630-ab2e-5b6a8d6410c9 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:17.987647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 208/408] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:17 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:17.994277 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565bcd9b-a705-4630-ab2e-5b6a8d6410c9 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:18.155258 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:18.157885 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c54d1da-aa6f-4968-a4a6-062ae562b48d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:18.183948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:18.195784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565bcd9b-a705-4630-ab2e-5b6a8d6410c9 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:18.217910 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e38ffb-9627-4942-8704-08a70ed55101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:18.218526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e38ffb-9627-4942-8704-08a70ed55101 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:18.218799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e38ffb-9627-4942-8704-08a70ed55101 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:18.219096 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e38ffb-9627-4942-8704-08a70ed55101 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:18.219923 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e38ffb-9627-4942-8704-08a70ed55101 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:18.220225 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e38ffb-9627-4942-8704-08a70ed55101 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:18.221598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e38ffb-9627-4942-8704-08a70ed55101 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:18.238527 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 201/409] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:18.244218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e38ffb-9627-4942-8704-08a70ed55101 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:18.257194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f54338e-add3-4ded-8559-be1fe1d66720 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:18.257888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f54338e-add3-4ded-8559-be1fe1d66720 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:18.258212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f54338e-add3-4ded-8559-be1fe1d66720 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:18.258486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f54338e-add3-4ded-8559-be1fe1d66720 None admin] PATH_INFO: `/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:18.265981 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f54338e-add3-4ded-8559-be1fe1d66720 None admin] RBAC: Authorizing `identity:get_group(group_id=cd7cfa9c290c45a3bcd6bf544e905891)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:18.268171 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f54338e-add3-4ded-8559-be1fe1d66720 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:18.269750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 209/410] 10.222.0.49 () {56 vars in 1180 bytes} [Fri Sep 16 03:27:17 2022] GET /identity/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891 => generated 226 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:18.279902 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f54338e-add3-4ded-8559-be1fe1d66720 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:18.444356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e38ffb-9627-4942-8704-08a70ed55101 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:18.481517 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f54338e-add3-4ded-8559-be1fe1d66720 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:18.510431 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e03e6c-b8d8-4a33-b8d5-829a1b7787e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:18.511097 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e03e6c-b8d8-4a33-b8d5-829a1b7787e3 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:18.511428 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e03e6c-b8d8-4a33-b8d5-829a1b7787e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:18.511700 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e03e6c-b8d8-4a33-b8d5-829a1b7787e3 None admin] PATH_INFO: `/v3/groups/88b3e22965ff4c70a94e3b19fa5b6ed6` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:18.512891 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e03e6c-b8d8-4a33-b8d5-829a1b7787e3 None admin] RBAC: Authorizing `identity:get_group(group_id=88b3e22965ff4c70a94e3b19fa5b6ed6)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:18.514328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e03e6c-b8d8-4a33-b8d5-829a1b7787e3 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:18.515906 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 202/411] 10.222.0.49 () {56 vars in 1180 bytes} [Fri Sep 16 03:27:18 2022] GET /identity/v3/groups/88b3e22965ff4c70a94e3b19fa5b6ed6 => generated 229 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:18.524321 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e03e6c-b8d8-4a33-b8d5-829a1b7787e3 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:18.551321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9eef56-7d8c-40ec-9cd2-3303b1a26400 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:18.552000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9eef56-7d8c-40ec-9cd2-3303b1a26400 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:18.552280 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9eef56-7d8c-40ec-9cd2-3303b1a26400 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:18.552549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9eef56-7d8c-40ec-9cd2-3303b1a26400 None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:18.554760 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9eef56-7d8c-40ec-9cd2-3303b1a26400 None admin] RBAC: Authorizing `identity:get_project(project_id=5f8ae3affd36481c8ce85ec30e7dedd9)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:18.556250 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9eef56-7d8c-40ec-9cd2-3303b1a26400 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:18.560455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 210/412] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:27:18 2022] GET /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9 => generated 302 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:18.577138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9eef56-7d8c-40ec-9cd2-3303b1a26400 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:18.584364 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:18.620267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:18.716346 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e03e6c-b8d8-4a33-b8d5-829a1b7787e3 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:18.753864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9eef56-7d8c-40ec-9cd2-3303b1a26400 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:18.784850 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430b3d72-2923-4707-844b-80154ddeac58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:18.786255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430b3d72-2923-4707-844b-80154ddeac58 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:18.787308 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430b3d72-2923-4707-844b-80154ddeac58 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:18.787764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430b3d72-2923-4707-844b-80154ddeac58 None admin] PATH_INFO: `/v3/projects/ccdd7a44ba5049fb939dad44d5bd406e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:18.789482 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430b3d72-2923-4707-844b-80154ddeac58 None admin] RBAC: Authorizing `identity:get_project(project_id=ccdd7a44ba5049fb939dad44d5bd406e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:18.791203 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430b3d72-2923-4707-844b-80154ddeac58 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:18.793948 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 203/413] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:27:18 2022] GET /identity/v3/projects/ccdd7a44ba5049fb939dad44d5bd406e => generated 344 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:18.795379 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b0c6ed-a225-4bb4-86c9-61baddc34dc2 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:18.795831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b0c6ed-a225-4bb4-86c9-61baddc34dc2 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:18.796289 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b0c6ed-a225-4bb4-86c9-61baddc34dc2 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:18.802600 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 204/414] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:18.803683 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b0c6ed-a225-4bb4-86c9-61baddc34dc2 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:18.809985 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad68862a-71fd-4cea-8587-8a2ad3f15036 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:18.810727 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad68862a-71fd-4cea-8587-8a2ad3f15036 None admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:18.811106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad68862a-71fd-4cea-8587-8a2ad3f15036 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:18.811504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad68862a-71fd-4cea-8587-8a2ad3f15036 None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9/groups/cd7cfa9c290c45a3bcd6bf544e905891/roles/fa8bdf568182483aac8a14ef98a12a95` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:18.814137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad68862a-71fd-4cea-8587-8a2ad3f15036 None admin] RBAC: Authorizing `identity:create_grant(project_id=5f8ae3affd36481c8ce85ec30e7dedd9, group_id=cd7cfa9c290c45a3bcd6bf544e905891, role_id=fa8bdf568182483aac8a14ef98a12a95)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:18.815508 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad68862a-71fd-4cea-8587-8a2ad3f15036 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:18.881320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 211/415] 10.222.0.49 () {58 vars in 1442 bytes} [Fri Sep 16 03:27:18 2022] PUT /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9/groups/cd7cfa9c290c45a3bcd6bf544e905891/roles/fa8bdf568182483aac8a14ef98a12a95 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:27:18.882900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279073eb-d324-491a-8122-0a505c4b05df None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:18.883423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279073eb-d324-491a-8122-0a505c4b05df None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:18.883957 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279073eb-d324-491a-8122-0a505c4b05df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:18.905035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-279073eb-d324-491a-8122-0a505c4b05df None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:18.906720 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-279073eb-d324-491a-8122-0a505c4b05df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:18.927830 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:18.933055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b0c6ed-a225-4bb4-86c9-61baddc34dc2 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:18.979553 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c877a06-233d-4264-83ba-1d5685a8b408 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:18.980187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c877a06-233d-4264-83ba-1d5685a8b408 None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:18.980466 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c877a06-233d-4264-83ba-1d5685a8b408 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:18.980737 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c877a06-233d-4264-83ba-1d5685a8b408 None admin] PATH_INFO: `/v3/projects/ccdd7a44ba5049fb939dad44d5bd406e/groups/88b3e22965ff4c70a94e3b19fa5b6ed6/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:18.984136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c877a06-233d-4264-83ba-1d5685a8b408 None admin] RBAC: Authorizing `identity:create_grant(project_id=ccdd7a44ba5049fb939dad44d5bd406e, group_id=88b3e22965ff4c70a94e3b19fa5b6ed6, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:18.984473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c877a06-233d-4264-83ba-1d5685a8b408 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:19.005263 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 212/416] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:19.061294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 205/417] 10.222.0.49 () {58 vars in 1442 bytes} [Fri Sep 16 03:27:18 2022] PUT /identity/v3/projects/ccdd7a44ba5049fb939dad44d5bd406e/groups/88b3e22965ff4c70a94e3b19fa5b6ed6/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:27:19.291720 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279073eb-d324-491a-8122-0a505c4b05df None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:19.320778 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:19.418802 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279073eb-d324-491a-8122-0a505c4b05df None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:19.470742 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733d3af1-d168-4f69-b333-a9654d1aa19b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:19.471409 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733d3af1-d168-4f69-b333-a9654d1aa19b None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:19.471692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733d3af1-d168-4f69-b333-a9654d1aa19b None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:19.471965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733d3af1-d168-4f69-b333-a9654d1aa19b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:19.474756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3c5476-a37e-4f17-81d3-dbbb69dbdfb2 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:19.475070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3c5476-a37e-4f17-81d3-dbbb69dbdfb2 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:19.475347 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3c5476-a37e-4f17-81d3-dbbb69dbdfb2 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:19.476813 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 206/418] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:19.482018 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7403582-a4ce-445e-adc6-6fc32334469c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:19.482550 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7403582-a4ce-445e-adc6-6fc32334469c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:19.483231 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7403582-a4ce-445e-adc6-6fc32334469c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:19.491823 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733d3af1-d168-4f69-b333-a9654d1aa19b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:19.496901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-733d3af1-d168-4f69-b333-a9654d1aa19b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:27:19.496901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:19.496901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:19.496901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:19.496901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:19.496901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:19.496901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:19.496901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:19.496901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:19.496901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:19.496901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:19.496901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:19.496901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:19.496901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:19.496901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:19.496901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:19.497813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:19.497813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:19.497813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:19.497813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:19.497813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:19.497813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:19.497813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:19.497813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:19.497813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:19.497813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:19.497813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:19.497813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:19.497813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:19.497813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:19.497813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:19.497813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:19.498656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:19.498656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:19.498656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:19.498656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:19.498656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:19.498656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:19.498656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:19.498656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:19.498656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:19.498656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:19.498656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:27:19.498656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:19.498656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 213/419] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:19 2022] GET /identity/v3/roles/member => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:19.507782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733d3af1-d168-4f69-b333-a9654d1aa19b None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:19.511204 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-d7403582-a4ce-445e-adc6-6fc32334469c None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:19.512621 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d7403582-a4ce-445e-adc6-6fc32334469c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:19.618445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:19.638297 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733d3af1-d168-4f69-b333-a9654d1aa19b None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:19.654995 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 207/420] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 03:27:19.690126 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09ae808-c17d-47ca-80fa-ae6913fd4bb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:19.690736 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09ae808-c17d-47ca-80fa-ae6913fd4bb7 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:19.691039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09ae808-c17d-47ca-80fa-ae6913fd4bb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:19.691309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09ae808-c17d-47ca-80fa-ae6913fd4bb7 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:19.691971 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09ae808-c17d-47ca-80fa-ae6913fd4bb7 None admin] RBAC: Adding query filter params (name=member) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:19.692269 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09ae808-c17d-47ca-80fa-ae6913fd4bb7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:19.693640 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09ae808-c17d-47ca-80fa-ae6913fd4bb7 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:19.707261 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 214/421] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:27:19 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:19.714992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7403582-a4ce-445e-adc6-6fc32334469c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:19.846725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:19.863109 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7403582-a4ce-445e-adc6-6fc32334469c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:19.909194 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e91bf8-4ac2-42a5-b3e7-310500a4da75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:19.909738 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e91bf8-4ac2-42a5-b3e7-310500a4da75 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:19.910018 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e91bf8-4ac2-42a5-b3e7-310500a4da75 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:19.910286 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e91bf8-4ac2-42a5-b3e7-310500a4da75 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:19.959335 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09ae808-c17d-47ca-80fa-ae6913fd4bb7 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:20.036832 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e91bf8-4ac2-42a5-b3e7-310500a4da75 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:20.038658 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e91bf8-4ac2-42a5-b3e7-310500a4da75 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:20.144515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09ae808-c17d-47ca-80fa-ae6913fd4bb7 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:20.177990 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:20.188087 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ed113a-f562-4e66-b7fa-0a63d59fd01a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:20.192403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ed113a-f562-4e66-b7fa-0a63d59fd01a None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:20.192403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ed113a-f562-4e66-b7fa-0a63d59fd01a None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:20.192403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ed113a-f562-4e66-b7fa-0a63d59fd01a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:20.197457 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 208/422] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 483 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:27:20.198815 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dd7475-e460-462f-8341-3c36dc668123 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:20.199125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dd7475-e460-462f-8341-3c36dc668123 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:20.199405 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dd7475-e460-462f-8341-3c36dc668123 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:20.200700 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 209/423] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:20.201485 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dd7475-e460-462f-8341-3c36dc668123 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:20.210952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ed113a-f562-4e66-b7fa-0a63d59fd01a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:20.212534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-71ed113a-f562-4e66-b7fa-0a63d59fd01a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:27:20.212534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:20.212534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:20.212534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:20.212534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:20.212534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:20.212534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:20.212534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:20.212534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:20.212534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:20.212534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:20.212534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:20.212534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:20.212534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:20.212534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:20.212534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:20.213429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:20.213429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:20.213429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:20.213429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:20.213429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:20.213429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:20.213429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:20.213429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:20.213429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:20.213429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:20.213429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:20.213429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:20.213429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:20.213429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:20.213429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:20.213429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:20.214304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:20.214304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:20.214304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:20.214304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:20.214304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:20.214304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:20.214304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:20.214304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:20.214304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:20.214304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:20.214304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 03:27:20.214304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:20.214926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 215/424] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:19 2022] GET /identity/v3/roles/member => generated 84 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:20.216161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d086ba6f-a6ab-4867-9454-bd27d707ac85 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:20.216576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d086ba6f-a6ab-4867-9454-bd27d707ac85 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:20.216983 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d086ba6f-a6ab-4867-9454-bd27d707ac85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:20.241523 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:20.243895 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-d086ba6f-a6ab-4867-9454-bd27d707ac85 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:20.245102 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d086ba6f-a6ab-4867-9454-bd27d707ac85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:20.321625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 216/425] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:20.323081 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d086ba6f-a6ab-4867-9454-bd27d707ac85 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:20.334178 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dd7475-e460-462f-8341-3c36dc668123 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:20.373105 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3102715-7b22-40e3-83a3-f255dd7e0b96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:20.373739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3102715-7b22-40e3-83a3-f255dd7e0b96 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:20.374092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3102715-7b22-40e3-83a3-f255dd7e0b96 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:20.374500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3102715-7b22-40e3-83a3-f255dd7e0b96 None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:20.377031 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3102715-7b22-40e3-83a3-f255dd7e0b96 None admin] RBAC: Authorizing `identity:get_project(project_id=8d00e05132b54e9b856788c06cdb7a56)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:20.378475 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3102715-7b22-40e3-83a3-f255dd7e0b96 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:20.380420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 210/426] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:27:20 2022] GET /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56 => generated 298 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:20.387184 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3102715-7b22-40e3-83a3-f255dd7e0b96 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:20.499265 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:20.504796 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d086ba6f-a6ab-4867-9454-bd27d707ac85 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:20.584474 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536c2766-6415-4b8c-ba36-ca691874f0f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:20.585106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536c2766-6415-4b8c-ba36-ca691874f0f5 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:20.585389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536c2766-6415-4b8c-ba36-ca691874f0f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:20.585666 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536c2766-6415-4b8c-ba36-ca691874f0f5 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:20.586340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-536c2766-6415-4b8c-ba36-ca691874f0f5 None admin] RBAC: Adding query filter params (name=member) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:20.586648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-536c2766-6415-4b8c-ba36-ca691874f0f5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:20.587295 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3102715-7b22-40e3-83a3-f255dd7e0b96 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:20.588076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-536c2766-6415-4b8c-ba36-ca691874f0f5 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:20.591212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:20.606897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 217/427] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:27:20 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:20.624970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536c2766-6415-4b8c-ba36-ca691874f0f5 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:20.633549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab0d828-c559-4899-9517-ea03eb13e325 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:20.634193 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab0d828-c559-4899-9517-ea03eb13e325 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:20.634471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab0d828-c559-4899-9517-ea03eb13e325 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:20.634745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab0d828-c559-4899-9517-ea03eb13e325 None admin] PATH_INFO: `/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:20.635887 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab0d828-c559-4899-9517-ea03eb13e325 None admin] RBAC: Authorizing `identity:get_group(group_id=cd7cfa9c290c45a3bcd6bf544e905891)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:20.637315 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab0d828-c559-4899-9517-ea03eb13e325 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:20.639192 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 211/428] 10.222.0.49 () {56 vars in 1180 bytes} [Fri Sep 16 03:27:20 2022] GET /identity/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891 => generated 226 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:20.645816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab0d828-c559-4899-9517-ea03eb13e325 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:20.773461 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536c2766-6415-4b8c-ba36-ca691874f0f5 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:20.774592 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab0d828-c559-4899-9517-ea03eb13e325 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:20.806228 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:20.810233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1e3493-7c74-4e07-a4bb-8781d0059e50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:20.810771 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1e3493-7c74-4e07-a4bb-8781d0059e50 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:20.811263 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1e3493-7c74-4e07-a4bb-8781d0059e50 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:20.811536 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1e3493-7c74-4e07-a4bb-8781d0059e50 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:20.819133 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66de4ff9-e053-497b-a150-a7fb2745c5ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:20.819918 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66de4ff9-e053-497b-a150-a7fb2745c5ad None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:20.820347 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66de4ff9-e053-497b-a150-a7fb2745c5ad None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:20.822134 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66de4ff9-e053-497b-a150-a7fb2745c5ad None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:20.822134 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66de4ff9-e053-497b-a150-a7fb2745c5ad None admin] RBAC: Adding query filter params (group.id=cd7cfa9c290c45a3bcd6bf544e905891, role.id=589e1f1333a042b7a00c53384a2f5426, scope.project.id=8d00e05132b54e9b856788c06cdb7a56) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:20.822134 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66de4ff9-e053-497b-a150-a7fb2745c5ad None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:20.823869 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66de4ff9-e053-497b-a150-a7fb2745c5ad None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:20.839073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 212/429] 10.222.0.49 () {56 vars in 1376 bytes} [Fri Sep 16 03:27:20 2022] GET /identity/v3/role_assignments?group.id=cd7cfa9c290c45a3bcd6bf544e905891&scope.project.id=8d00e05132b54e9b856788c06cdb7a56&role.id=589e1f1333a042b7a00c53384a2f5426 => generated 611 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:20.840079 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66de4ff9-e053-497b-a150-a7fb2745c5ad None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:20.921356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:20.938752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1e3493-7c74-4e07-a4bb-8781d0059e50 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:20.940477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1e3493-7c74-4e07-a4bb-8781d0059e50 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:21.077209 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66de4ff9-e053-497b-a150-a7fb2745c5ad None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:21.095995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 218/430] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 471 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 16 03:27:21.102159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1e3493-7c74-4e07-a4bb-8781d0059e50 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:21.120964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbdf39c-61dc-47d6-9fd8-70577bbb8c7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:21.121601 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbdf39c-61dc-47d6-9fd8-70577bbb8c7b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:21.121931 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbdf39c-61dc-47d6-9fd8-70577bbb8c7b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:21.122206 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbdf39c-61dc-47d6-9fd8-70577bbb8c7b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:21.133892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbdf39c-61dc-47d6-9fd8-70577bbb8c7b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:21.139852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-edbdf39c-61dc-47d6-9fd8-70577bbb8c7b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:27:21.139852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:21.139852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:21.139852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:21.139852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:21.139852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:21.139852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:21.139852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:21.139852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:21.139852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:21.139852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:21.139852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:21.139852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:21.139852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:21.139852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:21.139852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:21.140792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:21.140792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:21.140792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:21.140792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:21.140792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:21.140792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:21.140792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:21.140792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:21.140792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:21.140792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:21.140792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:21.140792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:21.140792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:21.140792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:21.140792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:21.140792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:21.142226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:21.142226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:21.142226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:21.142226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:21.142226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:21.142226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:21.142226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:21.142226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:21.142226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:21.142226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:21.142226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:27:21.142226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:21.142226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 213/431] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:27:20 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:21.147009 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbdf39c-61dc-47d6-9fd8-70577bbb8c7b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:21.202326 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:21.239172 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:21.247172 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1e3493-7c74-4e07-a4bb-8781d0059e50 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:21.269271 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbdf39c-61dc-47d6-9fd8-70577bbb8c7b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:21.291910 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0e2095-d80a-45e1-a9c2-dc04618fdc7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:21.292517 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0e2095-d80a-45e1-a9c2-dc04618fdc7d None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:21.292932 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0e2095-d80a-45e1-a9c2-dc04618fdc7d None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:21.293278 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0e2095-d80a-45e1-a9c2-dc04618fdc7d None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:21.294520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0e2095-d80a-45e1-a9c2-dc04618fdc7d None admin] RBAC: Authorizing `identity:get_project(project_id=5f8ae3affd36481c8ce85ec30e7dedd9)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:21.296104 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0e2095-d80a-45e1-a9c2-dc04618fdc7d None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:21.298652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 219/432] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:27:21 2022] GET /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9 => generated 302 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:21.312398 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0e2095-d80a-45e1-a9c2-dc04618fdc7d None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:21.322076 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e4dcc0-e6c2-49e8-a523-84a1e816d96f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:21.327789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e4dcc0-e6c2-49e8-a523-84a1e816d96f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:21.328036 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e4dcc0-e6c2-49e8-a523-84a1e816d96f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:21.328410 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e4dcc0-e6c2-49e8-a523-84a1e816d96f None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:21.330205 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e4dcc0-e6c2-49e8-a523-84a1e816d96f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:21.330482 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e4dcc0-e6c2-49e8-a523-84a1e816d96f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:21.332017 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e4dcc0-e6c2-49e8-a523-84a1e816d96f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:21.342598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 214/433] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:21 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:21.348350 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e4dcc0-e6c2-49e8-a523-84a1e816d96f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:21.449269 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0e2095-d80a-45e1-a9c2-dc04618fdc7d None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:21.493719 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e4dcc0-e6c2-49e8-a523-84a1e816d96f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:21.497846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df3ff72-b2af-4a9f-ab43-4d797b2bfc53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:21.498379 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df3ff72-b2af-4a9f-ab43-4d797b2bfc53 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:21.498724 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df3ff72-b2af-4a9f-ab43-4d797b2bfc53 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:21.499093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df3ff72-b2af-4a9f-ab43-4d797b2bfc53 None admin] PATH_INFO: `/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:21.500313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df3ff72-b2af-4a9f-ab43-4d797b2bfc53 None admin] RBAC: Authorizing `identity:get_group(group_id=cd7cfa9c290c45a3bcd6bf544e905891)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:21.502467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df3ff72-b2af-4a9f-ab43-4d797b2bfc53 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:21.504896 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 220/434] 10.222.0.49 () {56 vars in 1180 bytes} [Fri Sep 16 03:27:21 2022] GET /identity/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891 => generated 226 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:21.513179 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:21.513927 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df3ff72-b2af-4a9f-ab43-4d797b2bfc53 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:21.548443 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2885215-0922-407c-aaae-96586be01e5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:21.549514 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2885215-0922-407c-aaae-96586be01e5c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:21.549989 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2885215-0922-407c-aaae-96586be01e5c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:21.550478 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2885215-0922-407c-aaae-96586be01e5c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:21.555085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:21.630450 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2885215-0922-407c-aaae-96586be01e5c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:21.633433 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2885215-0922-407c-aaae-96586be01e5c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:21.636703 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df3ff72-b2af-4a9f-ab43-4d797b2bfc53 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:21.676728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d2c9fb-cd29-4a48-a851-f4a0e70263e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:21.677341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d2c9fb-cd29-4a48-a851-f4a0e70263e6 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:21.677625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d2c9fb-cd29-4a48-a851-f4a0e70263e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:21.677902 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d2c9fb-cd29-4a48-a851-f4a0e70263e6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:21.678638 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d2c9fb-cd29-4a48-a851-f4a0e70263e6 None admin] RBAC: Adding query filter params (group.id=cd7cfa9c290c45a3bcd6bf544e905891, role.id=589e1f1333a042b7a00c53384a2f5426, scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:21.678976 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d2c9fb-cd29-4a48-a851-f4a0e70263e6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:21.680410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d2c9fb-cd29-4a48-a851-f4a0e70263e6 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:21.686119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 221/435] 10.222.0.49 () {56 vars in 1376 bytes} [Fri Sep 16 03:27:21 2022] GET /identity/v3/role_assignments?group.id=cd7cfa9c290c45a3bcd6bf544e905891&scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9&role.id=589e1f1333a042b7a00c53384a2f5426 => generated 611 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:21.687432 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cc0c81-3b70-42ee-8283-7ca4b47f0a7a None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:21.687725 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cc0c81-3b70-42ee-8283-7ca4b47f0a7a None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:21.688005 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cc0c81-3b70-42ee-8283-7ca4b47f0a7a None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:21.689654 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 222/436] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:21.690761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc39d1b-95a8-4703-bfb4-99d3a4e6835e None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:21.691082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc39d1b-95a8-4703-bfb4-99d3a4e6835e None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:21.691357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc39d1b-95a8-4703-bfb4-99d3a4e6835e None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:21.692784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 223/437] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:21.698336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385d348f-0250-4d48-9713-05592e26e553 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:21.699137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385d348f-0250-4d48-9713-05592e26e553 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:21.699417 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385d348f-0250-4d48-9713-05592e26e553 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:21.721043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-385d348f-0250-4d48-9713-05592e26e553 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:21.722284 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-385d348f-0250-4d48-9713-05592e26e553 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:21.751435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 215/438] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:27:21.752928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55bf85e-d5d1-471d-a4de-a211d233cb34 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:21.753218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55bf85e-d5d1-471d-a4de-a211d233cb34 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:21.753497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55bf85e-d5d1-471d-a4de-a211d233cb34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:21.772496 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-b55bf85e-d5d1-471d-a4de-a211d233cb34 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:21.773741 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b55bf85e-d5d1-471d-a4de-a211d233cb34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:21.776863 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:21.804202 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 224/439] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:21.805375 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385d348f-0250-4d48-9713-05592e26e553 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:21.860857 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:21.864919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 216/440] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:21.953076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385d348f-0250-4d48-9713-05592e26e553 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:21.978110 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff69ff63-ae71-4bd9-9418-49f86af9eb51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:21.978744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff69ff63-ae71-4bd9-9418-49f86af9eb51 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:21.979063 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff69ff63-ae71-4bd9-9418-49f86af9eb51 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:21.979337 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff69ff63-ae71-4bd9-9418-49f86af9eb51 None admin] PATH_INFO: `/v3/projects/8d00e05132b54e9b856788c06cdb7a56` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:21.980453 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff69ff63-ae71-4bd9-9418-49f86af9eb51 None admin] RBAC: Authorizing `identity:get_project(project_id=8d00e05132b54e9b856788c06cdb7a56)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:21.981903 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff69ff63-ae71-4bd9-9418-49f86af9eb51 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:21.983530 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 225/441] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:27:21 2022] GET /identity/v3/projects/8d00e05132b54e9b856788c06cdb7a56 => generated 298 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:21.990131 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55bf85e-d5d1-471d-a4de-a211d233cb34 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:22.081675 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff69ff63-ae71-4bd9-9418-49f86af9eb51 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:22.091475 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55bf85e-d5d1-471d-a4de-a211d233cb34 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:22.128964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1d6e43-7bab-4232-ad02-f83dd4f0f104 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:22.129606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1d6e43-7bab-4232-ad02-f83dd4f0f104 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:22.130055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1d6e43-7bab-4232-ad02-f83dd4f0f104 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:22.130336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1d6e43-7bab-4232-ad02-f83dd4f0f104 None admin] PATH_INFO: `/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:22.131876 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1d6e43-7bab-4232-ad02-f83dd4f0f104 None admin] RBAC: Authorizing `identity:get_group(group_id=cd7cfa9c290c45a3bcd6bf544e905891)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:22.133306 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1d6e43-7bab-4232-ad02-f83dd4f0f104 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:22.135046 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 217/442] 10.222.0.49 () {56 vars in 1180 bytes} [Fri Sep 16 03:27:21 2022] GET /identity/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891 => generated 226 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:22.139172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1d6e43-7bab-4232-ad02-f83dd4f0f104 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:22.150037 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:22.179085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:22.185551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff69ff63-ae71-4bd9-9418-49f86af9eb51 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:22.220288 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662244fe-a963-4310-a29c-68064a3d553a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:22.220837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662244fe-a963-4310-a29c-68064a3d553a None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:22.221119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662244fe-a963-4310-a29c-68064a3d553a None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:22.221391 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662244fe-a963-4310-a29c-68064a3d553a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:22.231228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662244fe-a963-4310-a29c-68064a3d553a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:22.236203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-662244fe-a963-4310-a29c-68064a3d553a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:27:22.236203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:22.236203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:22.236203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:22.236203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:22.236203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:22.236203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:22.236203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:22.236203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:22.236203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:22.236203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:22.236203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:22.236203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:22.236203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:22.236203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:22.236203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:22.237305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:22.237305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:22.237305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:22.237305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:22.237305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:22.237305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:22.237305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:22.237305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:22.237305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:22.237305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:22.237305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:22.237305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:22.237305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:22.237305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:22.237305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:22.237305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:22.238477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:22.238477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:22.238477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:22.238477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:22.238477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:22.238477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:22.238477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:22.238477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:22.238477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:22.238477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:22.238477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:27:22.238477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:22.239783 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1d6e43-7bab-4232-ad02-f83dd4f0f104 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:22.239874 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 226/443] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:27:22 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:27:22.239874 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662244fe-a963-4310-a29c-68064a3d553a None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:22.271771 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddb5d52-4e14-4943-9a9c-5142b7caec60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:22.272398 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddb5d52-4e14-4943-9a9c-5142b7caec60 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:22.272673 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddb5d52-4e14-4943-9a9c-5142b7caec60 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:22.272940 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddb5d52-4e14-4943-9a9c-5142b7caec60 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:22.283970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddb5d52-4e14-4943-9a9c-5142b7caec60 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:22.288751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-8ddb5d52-4e14-4943-9a9c-5142b7caec60 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:27:22.288751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:22.288751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:22.288751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:22.288751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:22.288751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:22.288751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:22.288751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:22.288751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:22.288751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:22.288751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:22.288751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:22.288751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:22.288751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:22.288751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:22.288751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:22.289835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:22.289835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:22.289835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:22.289835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:22.289835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:22.289835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:22.289835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:22.289835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:22.289835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:22.289835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:22.289835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:22.289835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:22.289835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:22.289835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:22.289835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:22.289835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:22.290998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:22.290998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:22.290998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:22.290998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:22.290998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:22.290998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:22.290998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:22.290998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:22.290998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:22.290998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:22.290998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:27:22.290998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:22.290998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 218/444] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:27:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:27:22.290998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddb5d52-4e14-4943-9a9c-5142b7caec60 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:22.339063 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662244fe-a963-4310-a29c-68064a3d553a None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:22.359209 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:22.372682 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa2aaf5-e6a3-40e9-8763-fdb77cd137f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:22.373281 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa2aaf5-e6a3-40e9-8763-fdb77cd137f7 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:22.373560 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa2aaf5-e6a3-40e9-8763-fdb77cd137f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:22.373831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa2aaf5-e6a3-40e9-8763-fdb77cd137f7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:22.374628 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa2aaf5-e6a3-40e9-8763-fdb77cd137f7 None admin] RBAC: Adding query filter params (group.id=cd7cfa9c290c45a3bcd6bf544e905891, role.id=fa8bdf568182483aac8a14ef98a12a95, scope.project.id=8d00e05132b54e9b856788c06cdb7a56) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:22.374962 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa2aaf5-e6a3-40e9-8763-fdb77cd137f7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:22.376524 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa2aaf5-e6a3-40e9-8763-fdb77cd137f7 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:22.382448 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 227/445] 10.222.0.49 () {56 vars in 1376 bytes} [Fri Sep 16 03:27:22 2022] GET /identity/v3/role_assignments?group.id=cd7cfa9c290c45a3bcd6bf544e905891&scope.project.id=8d00e05132b54e9b856788c06cdb7a56&role.id=fa8bdf568182483aac8a14ef98a12a95 => generated 611 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:22.383392 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa2aaf5-e6a3-40e9-8763-fdb77cd137f7 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:22.394743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddb5d52-4e14-4943-9a9c-5142b7caec60 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:22.408664 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:22.435794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6e6103-33c4-4cda-9fce-23c3f771b14c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:22.436429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6e6103-33c4-4cda-9fce-23c3f771b14c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:22.436708 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6e6103-33c4-4cda-9fce-23c3f771b14c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:22.436980 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6e6103-33c4-4cda-9fce-23c3f771b14c None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:22.437658 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6e6103-33c4-4cda-9fce-23c3f771b14c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:22.437951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6e6103-33c4-4cda-9fce-23c3f771b14c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:22.439911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6e6103-33c4-4cda-9fce-23c3f771b14c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:22.449839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 219/446] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:22 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:22.455901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6e6103-33c4-4cda-9fce-23c3f771b14c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:22.504393 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa2aaf5-e6a3-40e9-8763-fdb77cd137f7 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:22.545458 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2957c0-417a-4bb4-a040-a083580f8ac4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:22.546076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2957c0-417a-4bb4-a040-a083580f8ac4 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:22.546380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2957c0-417a-4bb4-a040-a083580f8ac4 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:22.546629 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2957c0-417a-4bb4-a040-a083580f8ac4 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:22.547329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2957c0-417a-4bb4-a040-a083580f8ac4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:22.548487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2957c0-417a-4bb4-a040-a083580f8ac4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:22.549729 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2957c0-417a-4bb4-a040-a083580f8ac4 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:22.559313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 228/447] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:22.564938 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2957c0-417a-4bb4-a040-a083580f8ac4 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:22.569221 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6e6103-33c4-4cda-9fce-23c3f771b14c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:22.604057 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:22.612301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470bcbc6-4343-49af-a6a9-27e6247903aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:22.612917 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470bcbc6-4343-49af-a6a9-27e6247903aa None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:22.613416 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470bcbc6-4343-49af-a6a9-27e6247903aa None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:22.613523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470bcbc6-4343-49af-a6a9-27e6247903aa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:22.659595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:22.682360 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2957c0-417a-4bb4-a040-a083580f8ac4 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:22.702984 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470bcbc6-4343-49af-a6a9-27e6247903aa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:22.704990 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470bcbc6-4343-49af-a6a9-27e6247903aa None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:22.724523 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a573ac6-b1e7-42d9-9d19-32ed22d4201e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:22.725154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a573ac6-b1e7-42d9-9d19-32ed22d4201e None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:22.725420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a573ac6-b1e7-42d9-9d19-32ed22d4201e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:22.725691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a573ac6-b1e7-42d9-9d19-32ed22d4201e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:22.811725 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a573ac6-b1e7-42d9-9d19-32ed22d4201e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:22.814803 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a573ac6-b1e7-42d9-9d19-32ed22d4201e None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:22.826729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 220/448] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 16 03:27:22.836344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470bcbc6-4343-49af-a6a9-27e6247903aa None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:22.874525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:22.915299 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:22.947013 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 229/449] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:27:22.952519 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a573ac6-b1e7-42d9-9d19-32ed22d4201e None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:22.961788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470bcbc6-4343-49af-a6a9-27e6247903aa None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:22.995449 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5380c3-7406-44d2-9bd1-e5ae65c44f6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:22.996079 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5380c3-7406-44d2-9bd1-e5ae65c44f6b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:22.996357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5380c3-7406-44d2-9bd1-e5ae65c44f6b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:22.996632 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5380c3-7406-44d2-9bd1-e5ae65c44f6b None admin] PATH_INFO: `/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:22.997745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5380c3-7406-44d2-9bd1-e5ae65c44f6b None admin] RBAC: Authorizing `identity:get_project(project_id=5f8ae3affd36481c8ce85ec30e7dedd9)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:22.999214 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5380c3-7406-44d2-9bd1-e5ae65c44f6b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:23.001403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 221/450] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:27:22 2022] GET /identity/v3/projects/5f8ae3affd36481c8ce85ec30e7dedd9 => generated 302 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:23.006857 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5380c3-7406-44d2-9bd1-e5ae65c44f6b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:23.049689 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a573ac6-b1e7-42d9-9d19-32ed22d4201e None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:23.084739 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c525657c-6eb4-4545-be0e-c9f4115dc8b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:23.085309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c525657c-6eb4-4545-be0e-c9f4115dc8b8 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:23.085592 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c525657c-6eb4-4545-be0e-c9f4115dc8b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:23.085870 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c525657c-6eb4-4545-be0e-c9f4115dc8b8 None admin] PATH_INFO: `/v3/projects/ccdd7a44ba5049fb939dad44d5bd406e` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:23.087025 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c525657c-6eb4-4545-be0e-c9f4115dc8b8 None admin] RBAC: Authorizing `identity:get_project(project_id=ccdd7a44ba5049fb939dad44d5bd406e)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:23.088464 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c525657c-6eb4-4545-be0e-c9f4115dc8b8 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:23.090072 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 230/451] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:27:22 2022] GET /identity/v3/projects/ccdd7a44ba5049fb939dad44d5bd406e => generated 344 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:23.095511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c525657c-6eb4-4545-be0e-c9f4115dc8b8 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:23.100096 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:23.104666 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5380c3-7406-44d2-9bd1-e5ae65c44f6b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:23.139565 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb2ba4e-6e62-4b58-87a8-ae1c43ece35e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:23.140202 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb2ba4e-6e62-4b58-87a8-ae1c43ece35e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:23.140481 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb2ba4e-6e62-4b58-87a8-ae1c43ece35e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:23.140756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb2ba4e-6e62-4b58-87a8-ae1c43ece35e None admin] PATH_INFO: `/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:23.142216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb2ba4e-6e62-4b58-87a8-ae1c43ece35e None admin] RBAC: Authorizing `identity:get_group(group_id=cd7cfa9c290c45a3bcd6bf544e905891)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:23.142525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:23.143702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb2ba4e-6e62-4b58-87a8-ae1c43ece35e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:23.145282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 222/452] 10.222.0.49 () {56 vars in 1180 bytes} [Fri Sep 16 03:27:23 2022] GET /identity/v3/groups/cd7cfa9c290c45a3bcd6bf544e905891 => generated 226 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:23.150962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb2ba4e-6e62-4b58-87a8-ae1c43ece35e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:23.195320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c525657c-6eb4-4545-be0e-c9f4115dc8b8 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:23.229185 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7283b8f5-60f4-4268-9e7b-96676b65fb27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:23.229813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7283b8f5-60f4-4268-9e7b-96676b65fb27 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:23.230094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7283b8f5-60f4-4268-9e7b-96676b65fb27 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:23.230376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7283b8f5-60f4-4268-9e7b-96676b65fb27 None admin] PATH_INFO: `/v3/groups/88b3e22965ff4c70a94e3b19fa5b6ed6` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:23.231517 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7283b8f5-60f4-4268-9e7b-96676b65fb27 None admin] RBAC: Authorizing `identity:get_group(group_id=88b3e22965ff4c70a94e3b19fa5b6ed6)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:23.232970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7283b8f5-60f4-4268-9e7b-96676b65fb27 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:23.234524 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 231/453] 10.222.0.49 () {56 vars in 1180 bytes} [Fri Sep 16 03:27:23 2022] GET /identity/v3/groups/88b3e22965ff4c70a94e3b19fa5b6ed6 => generated 229 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:23.240807 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7283b8f5-60f4-4268-9e7b-96676b65fb27 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:23.259776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb2ba4e-6e62-4b58-87a8-ae1c43ece35e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:23.294260 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602fa733-b0ec-4c35-8efc-aaaf36b1f3f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:23.294863 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602fa733-b0ec-4c35-8efc-aaaf36b1f3f5 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:23.295176 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602fa733-b0ec-4c35-8efc-aaaf36b1f3f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:23.295451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602fa733-b0ec-4c35-8efc-aaaf36b1f3f5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:23.296170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602fa733-b0ec-4c35-8efc-aaaf36b1f3f5 None admin] RBAC: Adding query filter params (group.id=cd7cfa9c290c45a3bcd6bf544e905891, role.id=fa8bdf568182483aac8a14ef98a12a95, scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:23.296476 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602fa733-b0ec-4c35-8efc-aaaf36b1f3f5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:23.297943 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602fa733-b0ec-4c35-8efc-aaaf36b1f3f5 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:23.303549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 223/454] 10.222.0.49 () {56 vars in 1376 bytes} [Fri Sep 16 03:27:23 2022] GET /identity/v3/role_assignments?group.id=cd7cfa9c290c45a3bcd6bf544e905891&scope.project.id=5f8ae3affd36481c8ce85ec30e7dedd9&role.id=fa8bdf568182483aac8a14ef98a12a95 => generated 611 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:23.345956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7283b8f5-60f4-4268-9e7b-96676b65fb27 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:23.364870 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:23.383107 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18714c17-a065-4c34-a1d5-1e18fac36044 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:23.383721 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18714c17-a065-4c34-a1d5-1e18fac36044 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:23.383998 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18714c17-a065-4c34-a1d5-1e18fac36044 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:23.384268 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18714c17-a065-4c34-a1d5-1e18fac36044 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:23.384993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18714c17-a065-4c34-a1d5-1e18fac36044 None admin] RBAC: Adding query filter params (group.id=88b3e22965ff4c70a94e3b19fa5b6ed6, role.id=89c4b3a971224d468d33d2dc411230a2, scope.project.id=ccdd7a44ba5049fb939dad44d5bd406e) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:23.385296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18714c17-a065-4c34-a1d5-1e18fac36044 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:23.386758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18714c17-a065-4c34-a1d5-1e18fac36044 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:23.393563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 232/455] 10.222.0.49 () {56 vars in 1376 bytes} [Fri Sep 16 03:27:23 2022] GET /identity/v3/role_assignments?group.id=88b3e22965ff4c70a94e3b19fa5b6ed6&scope.project.id=ccdd7a44ba5049fb939dad44d5bd406e&role.id=89c4b3a971224d468d33d2dc411230a2 => generated 611 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:26.041621 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b2222a-5203-4dee-867a-7ddaed84f873 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:26.042180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b2222a-5203-4dee-867a-7ddaed84f873 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:26.042180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b2222a-5203-4dee-867a-7ddaed84f873 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:26.043938 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 224/456] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:26.049719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2368f5-0de1-48c1-8318-18f529e6fa7b None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:26.050244 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2368f5-0de1-48c1-8318-18f529e6fa7b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:26.050244 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2368f5-0de1-48c1-8318-18f529e6fa7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:26.075963 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-cb2368f5-0de1-48c1-8318-18f529e6fa7b None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:26.077166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-cb2368f5-0de1-48c1-8318-18f529e6fa7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:26.161724 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 233/457] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 03:27:26.167002 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e392f113-bab1-4d67-a2d3-871097e5ec89 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:26.167531 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e392f113-bab1-4d67-a2d3-871097e5ec89 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:26.168004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e392f113-bab1-4d67-a2d3-871097e5ec89 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:26.169529 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 225/458] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:26.175579 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5557172-7f8d-4ecc-8cf2-ef81aae66242 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:26.175869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5557172-7f8d-4ecc-8cf2-ef81aae66242 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:26.176149 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5557172-7f8d-4ecc-8cf2-ef81aae66242 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:26.200955 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-e5557172-7f8d-4ecc-8cf2-ef81aae66242 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:26.202137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e5557172-7f8d-4ecc-8cf2-ef81aae66242 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:26.291107 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 234/459] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:26.433055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e392f113-bab1-4d67-a2d3-871097e5ec89 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:26.472849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:26.565347 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e392f113-bab1-4d67-a2d3-871097e5ec89 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:26.616617 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e558e2c-661c-4d2d-a7b8-15c5323596dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:26.618462 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e558e2c-661c-4d2d-a7b8-15c5323596dc None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:26.618744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e558e2c-661c-4d2d-a7b8-15c5323596dc None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:26.619046 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e558e2c-661c-4d2d-a7b8-15c5323596dc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:26.635736 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e558e2c-661c-4d2d-a7b8-15c5323596dc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:26.637171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e558e2c-661c-4d2d-a7b8-15c5323596dc None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:26.658063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-4e558e2c-661c-4d2d-a7b8-15c5323596dc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:26.658063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:26.658063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:26.658063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:26.658063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:26.658063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:26.658063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:26.658063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:26.658063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:26.658063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:26.658063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:26.658063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:26.658063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:26.658063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:26.658063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:26.658063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:26.659083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:26.659083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:26.659083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:26.659083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:26.659083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:26.659083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:26.659083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:26.659083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:26.659083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:26.659083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:26.659083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:26.659083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:26.659083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:26.659083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:26.659083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:26.659083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:26.659083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:26.659973 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:26.659973 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:26.659973 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:26.659973 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 226/460] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:26.667987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5557172-7f8d-4ecc-8cf2-ef81aae66242 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:26.681327 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e558e2c-661c-4d2d-a7b8-15c5323596dc None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:26.734313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:26.781041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:26.805892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e558e2c-661c-4d2d-a7b8-15c5323596dc None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:26.806627 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5557172-7f8d-4ecc-8cf2-ef81aae66242 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:26.853332 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08f3759-3ee4-4180-8dd9-335c7defe686 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:26.854749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08f3759-3ee4-4180-8dd9-335c7defe686 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:26.855282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08f3759-3ee4-4180-8dd9-335c7defe686 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:26.855941 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08f3759-3ee4-4180-8dd9-335c7defe686 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:26.856550 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335d0419-b70c-401d-8478-147e34b5028b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:26.857388 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335d0419-b70c-401d-8478-147e34b5028b None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:26.857667 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335d0419-b70c-401d-8478-147e34b5028b None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:26.857937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335d0419-b70c-401d-8478-147e34b5028b None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:26.858582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335d0419-b70c-401d-8478-147e34b5028b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:26.858882 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335d0419-b70c-401d-8478-147e34b5028b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:26.861191 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335d0419-b70c-401d-8478-147e34b5028b None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:26.870684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08f3759-3ee4-4180-8dd9-335c7defe686 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:26.873285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08f3759-3ee4-4180-8dd9-335c7defe686 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:26.879287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 235/461] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:26.880356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b776e0-9518-4fc4-aac7-dd43d759900f None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:26.880650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b776e0-9518-4fc4-aac7-dd43d759900f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:26.881147 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b776e0-9518-4fc4-aac7-dd43d759900f None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:26.882455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 236/462] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:26.887947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b776e0-9518-4fc4-aac7-dd43d759900f None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:26.896717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-e08f3759-3ee4-4180-8dd9-335c7defe686 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:26.896717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:26.896717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:26.896717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:26.896717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:26.896717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:26.896717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:26.896717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:26.896717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:26.896717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:26.896717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:26.896717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:26.896717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:26.896717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:26.896717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:26.896717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:26.897694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:26.897694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:26.897694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:26.897694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:26.897694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:26.897694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:26.897694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:26.897694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:26.897694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:26.897694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:26.897694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:26.897694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:26.897694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:26.897694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:26.897694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:26.897694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:26.897694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:26.898590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:26.898590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:26.898590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:26.900600 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 227/463] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:27:26.907345 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e75e11-d86b-47d9-a3d2-ce80206831fe None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:26.908663 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e75e11-d86b-47d9-a3d2-ce80206831fe None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:26.909519 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e75e11-d86b-47d9-a3d2-ce80206831fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:26.933063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-50e75e11-d86b-47d9-a3d2-ce80206831fe None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:26.934649 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-50e75e11-d86b-47d9-a3d2-ce80206831fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:26.997674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:27.002092 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b776e0-9518-4fc4-aac7-dd43d759900f None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:27.021527 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 228/464] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:27.023145 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e75e11-d86b-47d9-a3d2-ce80206831fe None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:27.049435 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f8e01c-add7-443a-a0cb-a8f834b26267 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:27.050055 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f8e01c-add7-443a-a0cb-a8f834b26267 None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:27.050613 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f8e01c-add7-443a-a0cb-a8f834b26267 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:27.050691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f8e01c-add7-443a-a0cb-a8f834b26267 None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:27.051514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f8e01c-add7-443a-a0cb-a8f834b26267 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:27.053933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f8e01c-add7-443a-a0cb-a8f834b26267 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:27.095455 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:27.133364 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e75e11-d86b-47d9-a3d2-ce80206831fe None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:27.158942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 237/465] 10.222.0.49 () {60 vars in 1132 bytes} [Fri Sep 16 03:27:26 2022] POST /identity/v3/users => generated 246 bytes in 268 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:27:27.158942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d924c963-fa19-40ae-9c1a-58434f052fe2 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:27.158942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d924c963-fa19-40ae-9c1a-58434f052fe2 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:27.158942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d924c963-fa19-40ae-9c1a-58434f052fe2 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:27.162928 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 238/466] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:27.165140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39083e77-0c1a-4e30-ad9b-4421449d4672 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:27.165436 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39083e77-0c1a-4e30-ad9b-4421449d4672 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:27.165717 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39083e77-0c1a-4e30-ad9b-4421449d4672 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:27.186676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-39083e77-0c1a-4e30-ad9b-4421449d4672 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:27.188458 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-39083e77-0c1a-4e30-ad9b-4421449d4672 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:27.226784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668f116b-5142-4822-bc00-42dc96590d98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:27.227470 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668f116b-5142-4822-bc00-42dc96590d98 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:27.227761 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668f116b-5142-4822-bc00-42dc96590d98 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:27.228019 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668f116b-5142-4822-bc00-42dc96590d98 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:27.228692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668f116b-5142-4822-bc00-42dc96590d98 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:27.228994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668f116b-5142-4822-bc00-42dc96590d98 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:27.230402 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668f116b-5142-4822-bc00-42dc96590d98 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:27.268858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 229/467] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:27.273087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668f116b-5142-4822-bc00-42dc96590d98 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:27.302493 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 239/468] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 03:27:27.310058 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39083e77-0c1a-4e30-ad9b-4421449d4672 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:27.424551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:27.441244 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39083e77-0c1a-4e30-ad9b-4421449d4672 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:27.445171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668f116b-5142-4822-bc00-42dc96590d98 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:27.472838 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:27.477890 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f9be03-6e25-45e2-a93f-c528efd8f845 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:27.478580 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f9be03-6e25-45e2-a93f-c528efd8f845 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:27.478867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f9be03-6e25-45e2-a93f-c528efd8f845 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:27.479166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f9be03-6e25-45e2-a93f-c528efd8f845 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:27.488053 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593571d3-2d72-46f6-8143-830c466f8b16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:27.489316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593571d3-2d72-46f6-8143-830c466f8b16 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:27.489635 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593571d3-2d72-46f6-8143-830c466f8b16 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:27.489912 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593571d3-2d72-46f6-8143-830c466f8b16 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:27.490570 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593571d3-2d72-46f6-8143-830c466f8b16 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:27.491149 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f9be03-6e25-45e2-a93f-c528efd8f845 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:27.493280 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593571d3-2d72-46f6-8143-830c466f8b16 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:27.493394 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f9be03-6e25-45e2-a93f-c528efd8f845 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:27.509082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-a0f9be03-6e25-45e2-a93f-c528efd8f845 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:27.509082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:27.509082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:27.509082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:27.509082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:27.509082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:27.509082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:27.509082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:27.509082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:27.509082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:27.509082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:27.509082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:27.509082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:27.509082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:27.509082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:27.509082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:27.510484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:27.510484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:27.510484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:27.510484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:27.510484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:27.510484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:27.510484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:27.510484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:27.510484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:27.510484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:27.510484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:27.510484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:27.510484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:27.510484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:27.510484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:27.510484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:27.510484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:27.511370 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:27.511370 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:27.511370 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:27.511370 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 240/469] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:27:27.517877 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f9be03-6e25-45e2-a93f-c528efd8f845 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:27.586130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 230/470] 10.222.0.49 () {60 vars in 1132 bytes} [Fri Sep 16 03:27:27 2022] POST /identity/v3/users => generated 248 bytes in 314 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:27:27.661283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f9be03-6e25-45e2-a93f-c528efd8f845 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:27.715553 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593571d3-2d72-46f6-8143-830c466f8b16 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:27.733679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ec3606-ea5b-41e2-8e1d-99235f46d3d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:27.734319 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ec3606-ea5b-41e2-8e1d-99235f46d3d4 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:27.734602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ec3606-ea5b-41e2-8e1d-99235f46d3d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:27.734876 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ec3606-ea5b-41e2-8e1d-99235f46d3d4 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:27.736453 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ec3606-ea5b-41e2-8e1d-99235f46d3d4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:27.736869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ec3606-ea5b-41e2-8e1d-99235f46d3d4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:27.738367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ec3606-ea5b-41e2-8e1d-99235f46d3d4 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:27.751272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:27.766142 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 241/471] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:27.775987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ec3606-ea5b-41e2-8e1d-99235f46d3d4 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:27.883581 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593571d3-2d72-46f6-8143-830c466f8b16 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:27.915073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ec3606-ea5b-41e2-8e1d-99235f46d3d4 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:27.935210 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396be4b9-b79b-47ce-a0eb-8b0cabdeab87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:27.935875 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396be4b9-b79b-47ce-a0eb-8b0cabdeab87 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:27.936142 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396be4b9-b79b-47ce-a0eb-8b0cabdeab87 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:27.936424 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396be4b9-b79b-47ce-a0eb-8b0cabdeab87 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:27.951567 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396be4b9-b79b-47ce-a0eb-8b0cabdeab87 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:27.954496 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396be4b9-b79b-47ce-a0eb-8b0cabdeab87 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:27.962749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecd1abe-d9d1-42dc-8ffd-8c411e72852e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:27.963643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecd1abe-d9d1-42dc-8ffd-8c411e72852e None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:27.964139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecd1abe-d9d1-42dc-8ffd-8c411e72852e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:27.964639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecd1abe-d9d1-42dc-8ffd-8c411e72852e None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:27.965473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ecd1abe-d9d1-42dc-8ffd-8c411e72852e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:27.967519 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ecd1abe-d9d1-42dc-8ffd-8c411e72852e None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:27.973584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-396be4b9-b79b-47ce-a0eb-8b0cabdeab87 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:27.973584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:27.973584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:27.973584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:27.973584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:27.973584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:27.973584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:27.973584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:27.973584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:27.973584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:27.973584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:27.973584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:27.973584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:27.973584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:27.973584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:27.973584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:27.974569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:27.974569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:27.974569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:27.974569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:27.974569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:27.974569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:27.974569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:27.974569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:27.974569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:27.974569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:27.974569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:27.974569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:27.974569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:27.974569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:27.974569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:27.974569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:27.974569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:27.982356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:27.982356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:27.982356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:27.982356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 231/472] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:27.982356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f36f3e-0fdc-4977-a82e-4f05866ce330 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:27.982356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f36f3e-0fdc-4977-a82e-4f05866ce330 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:27.982356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f36f3e-0fdc-4977-a82e-4f05866ce330 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:27.982356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 232/473] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:27.982356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f36f3e-0fdc-4977-a82e-4f05866ce330 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:28.019312 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:28.095768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f36f3e-0fdc-4977-a82e-4f05866ce330 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:28.109190 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 242/474] 10.222.0.49 () {60 vars in 1132 bytes} [Fri Sep 16 03:27:27 2022] POST /identity/v3/users => generated 248 bytes in 334 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:27:28.112178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db325e9-1dbb-4958-806a-a0f220f079c6 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:28.112831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db325e9-1dbb-4958-806a-a0f220f079c6 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:28.113368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db325e9-1dbb-4958-806a-a0f220f079c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:28.140563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-8db325e9-1dbb-4958-806a-a0f220f079c6 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:28.143881 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-8db325e9-1dbb-4958-806a-a0f220f079c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:28.177558 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a25da6-adc6-4544-b9e7-be415f893193 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:28.179699 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a25da6-adc6-4544-b9e7-be415f893193 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:28.180517 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a25da6-adc6-4544-b9e7-be415f893193 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:28.181020 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a25da6-adc6-4544-b9e7-be415f893193 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:28.182770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a25da6-adc6-4544-b9e7-be415f893193 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:28.185496 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a25da6-adc6-4544-b9e7-be415f893193 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:28.188590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a25da6-adc6-4544-b9e7-be415f893193 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:28.218061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 233/475] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:28.224544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a25da6-adc6-4544-b9e7-be415f893193 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:28.264996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:28.328244 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 243/476] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:28.343975 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:28.354142 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a25da6-adc6-4544-b9e7-be415f893193 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:28.394647 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c123f51-b8bd-4b73-9d88-7f83b9e816cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:28.395657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c123f51-b8bd-4b73-9d88-7f83b9e816cb None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:28.396178 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c123f51-b8bd-4b73-9d88-7f83b9e816cb None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:28.396645 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c123f51-b8bd-4b73-9d88-7f83b9e816cb None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:28.397541 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c123f51-b8bd-4b73-9d88-7f83b9e816cb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:28.399277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c123f51-b8bd-4b73-9d88-7f83b9e816cb None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:28.472352 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 234/477] 10.222.0.49 () {60 vars in 1132 bytes} [Fri Sep 16 03:27:28 2022] POST /identity/v3/users => generated 249 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:27:28.633343 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db325e9-1dbb-4958-806a-a0f220f079c6 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:28.778860 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db325e9-1dbb-4958-806a-a0f220f079c6 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:28.828613 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c4baf8-a0af-4591-92aa-e9065b5364f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:28.829787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c4baf8-a0af-4591-92aa-e9065b5364f1 None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:28.830219 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c4baf8-a0af-4591-92aa-e9065b5364f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:28.830496 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c4baf8-a0af-4591-92aa-e9065b5364f1 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:28.831204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c4baf8-a0af-4591-92aa-e9065b5364f1 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:28.832593 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c4baf8-a0af-4591-92aa-e9065b5364f1 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:28.888709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.sql.core [None req-e0c4baf8-a0af-4591-92aa-e9065b5364f1 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 16 03:27:28.888709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 16 03:27:28.888709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [parameters: {'id': '9f66f121b7764ed78734b9aa36f259bb', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 16 03:27:28.888709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81761) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 16 03:27:28.907071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-e0c4baf8-a0af-4591-92aa-e9065b5364f1 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 16 03:27:28.907071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:28.907071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 16 03:27:28.907071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 16 03:27:28.907071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 16 03:27:28.907071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 16 03:27:28.907071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 16 03:27:28.907071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application result = self._query(query) Sep 16 03:27:28.907071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 16 03:27:28.907071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application conn.query(q) Sep 16 03:27:28.907071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 16 03:27:28.907071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 16 03:27:28.907071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 16 03:27:28.907071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application result.read() Sep 16 03:27:28.907071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 16 03:27:28.908154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 16 03:27:28.908154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 16 03:27:28.908154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 16 03:27:28.908154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 16 03:27:28.908154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 16 03:27:28.908154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 16 03:27:28.908154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 16 03:27:28.908154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 16 03:27:28.908154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:28.908154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 16 03:27:28.908154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:28.908154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:28.908154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 16 03:27:28.908154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 16 03:27:28.908154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 16 03:27:28.908154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return ref.to_dict() Sep 16 03:27:28.908154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 16 03:27:28.908154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application next(self.gen) Sep 16 03:27:28.909286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 16 03:27:28.909286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application yield resource Sep 16 03:27:28.909286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 16 03:27:28.909286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application next(self.gen) Sep 16 03:27:28.909286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 16 03:27:28.909286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self.session.rollback() Sep 16 03:27:28.909286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 16 03:27:28.909286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self.force_reraise() Sep 16 03:27:28.909286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 16 03:27:28.909286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise self.value Sep 16 03:27:28.909286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 16 03:27:28.909286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 16 03:27:28.909286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 16 03:27:28.909286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application session.commit() Sep 16 03:27:28.909286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 16 03:27:28.909286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 16 03:27:28.909286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 16 03:27:28.910356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._prepare_impl() Sep 16 03:27:28.910356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 16 03:27:28.910356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self.session.flush() Sep 16 03:27:28.910356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 16 03:27:28.910356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._flush(objects) Sep 16 03:27:28.910356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 16 03:27:28.910356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 16 03:27:28.910356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 16 03:27:28.910356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application compat.raise_( Sep 16 03:27:28.910356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 16 03:27:28.910356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception Sep 16 03:27:28.910356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 16 03:27:28.910356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application flush_context.execute() Sep 16 03:27:28.910356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 16 03:27:28.910356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rec.execute(self) Sep 16 03:27:28.910356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 16 03:27:28.910356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 16 03:27:28.911447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 16 03:27:28.911447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 16 03:27:28.911447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 16 03:27:28.911447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 16 03:27:28.911447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 16 03:27:28.911447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 16 03:27:28.911447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 16 03:27:28.911447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 16 03:27:28.911447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 16 03:27:28.911447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 16 03:27:28.911447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 16 03:27:28.911447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 16 03:27:28.911447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 16 03:27:28.911447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 16 03:27:28.911447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 16 03:27:28.911447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception Sep 16 03:27:28.912377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 16 03:27:28.912377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 16 03:27:28.912377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 16 03:27:28.912377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 16 03:27:28.912377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 16 03:27:28.912377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application result = self._query(query) Sep 16 03:27:28.912377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 16 03:27:28.912377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application conn.query(q) Sep 16 03:27:28.912377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 16 03:27:28.912377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 16 03:27:28.912377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 16 03:27:28.912377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application result.read() Sep 16 03:27:28.912377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 16 03:27:28.912377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 16 03:27:28.912377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 16 03:27:28.912377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 16 03:27:28.913154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 16 03:27:28.913154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 16 03:27:28.913154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 16 03:27:28.913154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 16 03:27:28.913154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 16 03:27:28.913154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 16 03:27:28.913154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [parameters: {'id': '9f66f121b7764ed78734b9aa36f259bb', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 16 03:27:28.913154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 16 03:27:28.913154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:28.913154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:27:28.913154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:28.913154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:28.913154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:28.913154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:28.913154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:28.913154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:28.913949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:28.913949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:28.913949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:28.913949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:28.913949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:28.913949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:28.913949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 16 03:27:28.913949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 16 03:27:28.913949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:28.913949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:28.913949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 16 03:27:28.913949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 16 03:27:28.913949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 16 03:27:28.913949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 16 03:27:28.913949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 16 03:27:28.913949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:28.916213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 244/478] 10.222.0.49 () {60 vars in 1131 bytes} [Fri Sep 16 03:27:28 2022] POST /identity/v3/roles => generated 144 bytes in 284 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:28.922115 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c123f51-b8bd-4b73-9d88-7f83b9e816cb None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:29.076437 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c123f51-b8bd-4b73-9d88-7f83b9e816cb None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:29.118465 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d47330-2951-40ae-9f2a-5a375b2168d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:29.119941 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d47330-2951-40ae-9f2a-5a375b2168d9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:29.120417 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d47330-2951-40ae-9f2a-5a375b2168d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:29.120901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d47330-2951-40ae-9f2a-5a375b2168d9 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:29.136953 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d47330-2951-40ae-9f2a-5a375b2168d9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:29.142994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-53d47330-2951-40ae-9f2a-5a375b2168d9 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:27:29.142994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:29.142994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:29.142994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:29.142994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:29.142994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:29.142994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:29.142994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:29.142994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:29.142994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:29.142994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:29.142994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:29.142994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:29.142994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:29.142994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:29.142994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:29.143890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:29.143890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:29.143890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:29.143890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:29.143890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:29.143890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:29.143890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:29.143890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:29.143890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:29.143890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:29.143890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:29.143890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:29.143890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:29.143890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:29.143890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:29.143890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:29.144695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:29.144695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:29.144695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:29.144695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:29.144695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:29.144695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:29.144695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:29.144695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:29.144695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:29.144695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:29.144695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 03:27:29.144695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:29.147294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 235/479] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:27:28 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:29.153223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c4baf8-a0af-4591-92aa-e9065b5364f1 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:29.216746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:29.284075 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c4baf8-a0af-4591-92aa-e9065b5364f1 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:29.340774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ad35ff-e64d-47ca-8643-9612829c5e45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:29.341428 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ad35ff-e64d-47ca-8643-9612829c5e45 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:29.341705 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ad35ff-e64d-47ca-8643-9612829c5e45 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:29.341976 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ad35ff-e64d-47ca-8643-9612829c5e45 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:29.342668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ad35ff-e64d-47ca-8643-9612829c5e45 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:29.343005 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ad35ff-e64d-47ca-8643-9612829c5e45 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:29.344492 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ad35ff-e64d-47ca-8643-9612829c5e45 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:29.358772 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 245/480] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:29 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:29.960004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedad8c0-ea91-434e-9448-f1addb8cc4aa None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:29.960398 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedad8c0-ea91-434e-9448-f1addb8cc4aa None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:29.960581 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedad8c0-ea91-434e-9448-f1addb8cc4aa None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:29.962153 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 236/481] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:29.967247 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b0dd6f-3771-433a-b04c-30cb1544aa49 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:29.967535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b0dd6f-3771-433a-b04c-30cb1544aa49 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:29.967803 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b0dd6f-3771-433a-b04c-30cb1544aa49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:29.991958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-d3b0dd6f-3771-433a-b04c-30cb1544aa49 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:29.993158 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d3b0dd6f-3771-433a-b04c-30cb1544aa49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:30.084055 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 246/482] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:30.247336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5542d782-292e-446f-921e-3bb1465b67e8 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:30.248289 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5542d782-292e-446f-921e-3bb1465b67e8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:30.248769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5542d782-292e-446f-921e-3bb1465b67e8 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:30.250603 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 237/483] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:30.256137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd31c5a-84ae-4f36-bbd9-e651ab5d9d39 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:30.256950 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd31c5a-84ae-4f36-bbd9-e651ab5d9d39 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:30.257441 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd31c5a-84ae-4f36-bbd9-e651ab5d9d39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:30.287947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-7dd31c5a-84ae-4f36-bbd9-e651ab5d9d39 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:30.289490 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-7dd31c5a-84ae-4f36-bbd9-e651ab5d9d39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:30.361499 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:30.403763 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 247/484] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:30.460731 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5542d782-292e-446f-921e-3bb1465b67e8 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:30.478772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:30.596254 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5542d782-292e-446f-921e-3bb1465b67e8 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:30.642772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28320e37-77de-4e81-8f62-7908525fd33e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:30.643774 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28320e37-77de-4e81-8f62-7908525fd33e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:30.644239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28320e37-77de-4e81-8f62-7908525fd33e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:30.644694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28320e37-77de-4e81-8f62-7908525fd33e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:30.661006 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28320e37-77de-4e81-8f62-7908525fd33e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:30.666130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-28320e37-77de-4e81-8f62-7908525fd33e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:30.666130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:30.666130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:30.666130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:30.666130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:30.666130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:30.666130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:30.666130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:30.666130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:30.666130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:30.666130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:30.666130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:30.666130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:30.666130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:30.666130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:30.666130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:30.671189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:30.671189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:30.671189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:30.671189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:30.671189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:30.671189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:30.671189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:30.671189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:30.671189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:30.671189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:30.671189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:30.671189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:30.671189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:30.671189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:30.671189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:30.671189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:30.672227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:30.672227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:30.672227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:30.672227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:30.672227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:30.672227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:30.672227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:30.672227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:30.672227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:30.672227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:30.672227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:30.672227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:30.674666 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 238/485] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:27:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:30.680133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd31c5a-84ae-4f36-bbd9-e651ab5d9d39 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:30.851208 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd31c5a-84ae-4f36-bbd9-e651ab5d9d39 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:30.890435 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a845be-67f9-42dc-8b1d-f46dc6d1b080 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:30.891079 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a845be-67f9-42dc-8b1d-f46dc6d1b080 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:30.891890 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a845be-67f9-42dc-8b1d-f46dc6d1b080 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:30.892166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a845be-67f9-42dc-8b1d-f46dc6d1b080 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:30.892923 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a845be-67f9-42dc-8b1d-f46dc6d1b080 None admin] RBAC: Adding query filter params (name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:30.893232 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a845be-67f9-42dc-8b1d-f46dc6d1b080 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:30.894619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a845be-67f9-42dc-8b1d-f46dc6d1b080 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:30.909129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28320e37-77de-4e81-8f62-7908525fd33e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:30.911275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 248/486] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:27:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:30.924742 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02f716f-996d-450b-bc0c-4190f99b8c7b None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:30.925030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02f716f-996d-450b-bc0c-4190f99b8c7b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:30.925296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02f716f-996d-450b-bc0c-4190f99b8c7b None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:30.926865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 249/487] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:27:30.933681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c246ed42-aa68-4b74-b3c2-2e1cb4e147fd None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:30.933965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c246ed42-aa68-4b74-b3c2-2e1cb4e147fd None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:30.934242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c246ed42-aa68-4b74-b3c2-2e1cb4e147fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:30.959602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-c246ed42-aa68-4b74-b3c2-2e1cb4e147fd None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:30.961118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c246ed42-aa68-4b74-b3c2-2e1cb4e147fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:30.996418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:31.004679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:31.047363 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 250/488] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:31.048286 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28320e37-77de-4e81-8f62-7908525fd33e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:31.048533 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c246ed42-aa68-4b74-b3c2-2e1cb4e147fd None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:31.103246 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d8b664-b4d6-4ed0-887c-a7c32a76ffb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:31.104243 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d8b664-b4d6-4ed0-887c-a7c32a76ffb6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:31.104805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d8b664-b4d6-4ed0-887c-a7c32a76ffb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:31.105281 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d8b664-b4d6-4ed0-887c-a7c32a76ffb6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:31.125708 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d8b664-b4d6-4ed0-887c-a7c32a76ffb6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:31.127653 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d8b664-b4d6-4ed0-887c-a7c32a76ffb6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:31.152500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-e5d8b664-b4d6-4ed0-887c-a7c32a76ffb6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:31.152500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:31.152500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:31.152500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:31.152500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:31.152500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:31.152500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:31.152500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:31.152500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:31.152500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:31.152500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:31.152500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:31.152500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:31.152500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:31.152500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:31.152500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:31.153452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:31.153452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:31.153452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:31.153452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:31.153452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:31.153452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:31.153452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:31.153452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:31.153452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:31.153452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:31.153452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:31.153452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:31.153452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:31.153452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:31.153452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:31.153452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:31.153452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:31.154302 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:31.154302 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:31.154302 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:31.157061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 239/489] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:31.158564 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd5042a-f61a-412e-aad7-f6273441f1a5 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:31.159084 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd5042a-f61a-412e-aad7-f6273441f1a5 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:31.159565 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd5042a-f61a-412e-aad7-f6273441f1a5 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:31.161489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 240/490] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:31.162646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd5042a-f61a-412e-aad7-f6273441f1a5 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:31.204353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c246ed42-aa68-4b74-b3c2-2e1cb4e147fd None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:31.242812 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be48217f-3538-4e18-8370-8cc85641d21c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:31.243993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be48217f-3538-4e18-8370-8cc85641d21c None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:31.245278 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be48217f-3538-4e18-8370-8cc85641d21c None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:31.245794 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be48217f-3538-4e18-8370-8cc85641d21c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:31.259842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be48217f-3538-4e18-8370-8cc85641d21c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:31.267272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-be48217f-3538-4e18-8370-8cc85641d21c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:31.267272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:31.267272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:31.267272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:31.267272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:31.267272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:31.267272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:31.267272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:31.267272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:31.267272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:31.267272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:31.267272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:31.267272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:31.267272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:31.267272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:31.267272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:31.268233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:31.268233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:31.268233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:31.268233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:31.268233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:31.268233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:31.268233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:31.268233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:31.268233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:31.268233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:31.268233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:31.268233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:31.268233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:31.268233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:31.268233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:31.268233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:31.269049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:31.269049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:31.269049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:31.269049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:31.269049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:31.269049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:31.269049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:31.269049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:31.269049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:31.269049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:31.269049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:31.269049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:31.276508 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 251/491] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:27:31 2022] GET /identity/v3/roles/service => generated 85 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:31.276956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f632552-6652-46e2-999d-606de53fdb3f None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:31.277206 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f632552-6652-46e2-999d-606de53fdb3f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:31.277529 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f632552-6652-46e2-999d-606de53fdb3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:31.287933 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd5042a-f61a-412e-aad7-f6273441f1a5 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:31.296490 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-2f632552-6652-46e2-999d-606de53fdb3f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:31.297738 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-2f632552-6652-46e2-999d-606de53fdb3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:31.318810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:31.320915 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:31.341952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a586ff-7069-46de-b84d-fce2f4e767c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:31.342569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a586ff-7069-46de-b84d-fce2f4e767c2 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:31.342850 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a586ff-7069-46de-b84d-fce2f4e767c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:31.343160 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a586ff-7069-46de-b84d-fce2f4e767c2 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:31.343802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a586ff-7069-46de-b84d-fce2f4e767c2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:31.344108 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a586ff-7069-46de-b84d-fce2f4e767c2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:31.346718 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a586ff-7069-46de-b84d-fce2f4e767c2 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:31.361109 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 241/492] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:31.361109 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a586ff-7069-46de-b84d-fce2f4e767c2 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:31.379763 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 252/493] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:31.380789 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f632552-6652-46e2-999d-606de53fdb3f None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:31.479015 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a586ff-7069-46de-b84d-fce2f4e767c2 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:31.497092 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f632552-6652-46e2-999d-606de53fdb3f None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:31.514337 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f946917-5c26-4a99-bc91-669fff8e3042 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:31.515117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f946917-5c26-4a99-bc91-669fff8e3042 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:31.515549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f946917-5c26-4a99-bc91-669fff8e3042 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:31.515925 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f946917-5c26-4a99-bc91-669fff8e3042 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:31.516770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f946917-5c26-4a99-bc91-669fff8e3042 None admin] RBAC: Adding query filter params (name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:31.517179 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f946917-5c26-4a99-bc91-669fff8e3042 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:31.518659 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f946917-5c26-4a99-bc91-669fff8e3042 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:31.532348 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 242/494] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:27:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:31.532348 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f946917-5c26-4a99-bc91-669fff8e3042 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:31.549655 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59919a0-93b8-46d3-aa7a-64ee3d31c564 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:31.550900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59919a0-93b8-46d3-aa7a-64ee3d31c564 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:31.551417 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59919a0-93b8-46d3-aa7a-64ee3d31c564 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:31.551837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59919a0-93b8-46d3-aa7a-64ee3d31c564 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:31.567991 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59919a0-93b8-46d3-aa7a-64ee3d31c564 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:31.573166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-c59919a0-93b8-46d3-aa7a-64ee3d31c564 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:31.573166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:31.573166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:31.573166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:31.573166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:31.573166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:31.573166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:31.573166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:31.573166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:31.573166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:31.573166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:31.573166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:31.573166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:31.573166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:31.573166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:31.573166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:31.574039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:31.574039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:31.574039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:31.574039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:31.574039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:31.574039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:31.574039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:31.574039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:31.574039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:31.574039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:31.574039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:31.574039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:31.574039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:31.574039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:31.574039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:31.574039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:31.574855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:31.574855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:31.574855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:31.574855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:31.574855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:31.574855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:31.574855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:31.574855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:31.574855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:31.574855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:31.574855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:31.574855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:31.574855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 253/495] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:27:31 2022] GET /identity/v3/roles/service => generated 85 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:31.576187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59919a0-93b8-46d3-aa7a-64ee3d31c564 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:31.589601 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:31.595736 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:31.666441 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f946917-5c26-4a99-bc91-669fff8e3042 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:31.686978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59919a0-93b8-46d3-aa7a-64ee3d31c564 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:31.704706 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c805af13-6b10-4cd2-bf2f-aaf2437512c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:31.705345 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c805af13-6b10-4cd2-bf2f-aaf2437512c9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:31.705624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c805af13-6b10-4cd2-bf2f-aaf2437512c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:31.705905 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c805af13-6b10-4cd2-bf2f-aaf2437512c9 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:31.713685 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c805af13-6b10-4cd2-bf2f-aaf2437512c9 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:31.715163 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c805af13-6b10-4cd2-bf2f-aaf2437512c9 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:31.728158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-c805af13-6b10-4cd2-bf2f-aaf2437512c9 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 16 03:27:31.728158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:31.728158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:31.728158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:31.728158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:31.728158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:31.728158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:31.728158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:31.728158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:31.728158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:31.728158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:31.728158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:31.728158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:27:31.728158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:27:31.728158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:27:31.728158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:27:31.729307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:31.729307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:31.729307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:27:31.729307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:31.729307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:27:31.729307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:31.729307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:31.729307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:31.729307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:31.729307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:31.729307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:31.729307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:31.729307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:31.729307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:31.729307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:31.729307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:31.729307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:31.729307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:31.730226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:31.730226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:31.730226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:27:31.730226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:27:31.730226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:27:31.730226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:27:31.730226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:27:31.730226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:27:31.730226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 16 03:27:31.730226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:31.730226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 243/496] 10.222.0.49 () {56 vars in 1093 bytes} [Fri Sep 16 03:27:31 2022] GET /identity/v3/users/nova => generated 82 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:31.730788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c805af13-6b10-4cd2-bf2f-aaf2437512c9 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:31.734441 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ce7a93-5ac4-40a6-bbe7-267d2e3369d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:31.735846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ce7a93-5ac4-40a6-bbe7-267d2e3369d1 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:31.736264 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ce7a93-5ac4-40a6-bbe7-267d2e3369d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:31.736672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ce7a93-5ac4-40a6-bbe7-267d2e3369d1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:31.752266 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ce7a93-5ac4-40a6-bbe7-267d2e3369d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:31.753730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ce7a93-5ac4-40a6-bbe7-267d2e3369d1 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:31.768575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-73ce7a93-5ac4-40a6-bbe7-267d2e3369d1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:31.768575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:31.768575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:31.768575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:31.768575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:31.768575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:31.768575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:31.768575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:31.768575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:31.768575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:31.768575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:31.768575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:31.768575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:31.768575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:31.768575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:31.768575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:31.769733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:31.769733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:31.769733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:31.769733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:31.769733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:31.769733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:31.769733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:31.769733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:31.769733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:31.769733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:31.769733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:31.769733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:31.769733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:31.769733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:31.769733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:31.769733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:31.769733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:31.770619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:31.770619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:31.770619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:31.770619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 254/497] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:31.770845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ce7a93-5ac4-40a6-bbe7-267d2e3369d1 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:31.841955 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:31.859749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c805af13-6b10-4cd2-bf2f-aaf2437512c9 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:31.861867 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:31.884096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ce7a93-5ac4-40a6-bbe7-267d2e3369d1 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:31.897644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aec3c52-3c2a-4ef8-915b-c20b74fbdc16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:31.898697 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aec3c52-3c2a-4ef8-915b-c20b74fbdc16 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:31.899008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aec3c52-3c2a-4ef8-915b-c20b74fbdc16 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:31.899306 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aec3c52-3c2a-4ef8-915b-c20b74fbdc16 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:31.899958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aec3c52-3c2a-4ef8-915b-c20b74fbdc16 None admin] RBAC: Adding query filter params (name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:31.900264 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aec3c52-3c2a-4ef8-915b-c20b74fbdc16 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:31.901644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aec3c52-3c2a-4ef8-915b-c20b74fbdc16 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:31.914727 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 244/498] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:27:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:31.914727 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aec3c52-3c2a-4ef8-915b-c20b74fbdc16 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:31.927507 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f61aead-464e-4ca5-a2d0-0a9555925692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:31.928330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f61aead-464e-4ca5-a2d0-0a9555925692 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:31.928763 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f61aead-464e-4ca5-a2d0-0a9555925692 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:31.929042 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f61aead-464e-4ca5-a2d0-0a9555925692 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:31.941011 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f61aead-464e-4ca5-a2d0-0a9555925692 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:31.947148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-0f61aead-464e-4ca5-a2d0-0a9555925692 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:31.947148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:31.947148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:31.947148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:31.947148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:31.947148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:31.947148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:31.947148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:31.947148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:31.947148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:31.947148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:31.947148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:31.947148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:31.947148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:31.947148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:31.947148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:31.948050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:31.948050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:31.948050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:31.948050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:31.948050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:31.948050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:31.948050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:31.948050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:31.948050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:31.948050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:31.948050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:31.948050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:31.948050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:31.948050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:31.948050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:31.948050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:31.948891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:31.948891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:31.948891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:31.948891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:31.948891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:31.948891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:31.948891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:31.948891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:31.948891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:31.948891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:31.948891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:31.948891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:31.948891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 255/499] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:27:31 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:31.949558 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f61aead-464e-4ca5-a2d0-0a9555925692 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:32.029309 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aec3c52-3c2a-4ef8-915b-c20b74fbdc16 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:32.061613 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f61aead-464e-4ca5-a2d0-0a9555925692 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:32.067815 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2e1783-1e58-4313-9553-cae9cd42275e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:32.068433 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2e1783-1e58-4313-9553-cae9cd42275e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:32.068712 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2e1783-1e58-4313-9553-cae9cd42275e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:32.068986 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2e1783-1e58-4313-9553-cae9cd42275e None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:32.069743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2e1783-1e58-4313-9553-cae9cd42275e None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:32.070050 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2e1783-1e58-4313-9553-cae9cd42275e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:32.071519 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2e1783-1e58-4313-9553-cae9cd42275e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:32.087755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:32.098293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6c1cb2-b20d-4614-b62b-473669df6a88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:32.099816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6c1cb2-b20d-4614-b62b-473669df6a88 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:32.100081 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6c1cb2-b20d-4614-b62b-473669df6a88 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:32.100368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6c1cb2-b20d-4614-b62b-473669df6a88 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:32.101004 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6c1cb2-b20d-4614-b62b-473669df6a88 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:32.101307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6c1cb2-b20d-4614-b62b-473669df6a88 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:32.102693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6c1cb2-b20d-4614-b62b-473669df6a88 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:32.111717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 245/500] 10.222.0.49 () {56 vars in 1133 bytes} [Fri Sep 16 03:27:31 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:32.119015 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 256/501] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:32.119015 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6c1cb2-b20d-4614-b62b-473669df6a88 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:32.119216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2e1783-1e58-4313-9553-cae9cd42275e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:32.145238 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:32.218413 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6c1cb2-b20d-4614-b62b-473669df6a88 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:32.220191 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2e1783-1e58-4313-9553-cae9cd42275e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:32.250122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46fb6c5-7b22-42f4-a97b-9d598590e3b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:32.251360 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46fb6c5-7b22-42f4-a97b-9d598590e3b3 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:32.251687 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46fb6c5-7b22-42f4-a97b-9d598590e3b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:32.252019 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46fb6c5-7b22-42f4-a97b-9d598590e3b3 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:32.252680 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46fb6c5-7b22-42f4-a97b-9d598590e3b3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:32.252937 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bf7b4f-d031-4dbb-b071-653330e479d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:32.253047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46fb6c5-7b22-42f4-a97b-9d598590e3b3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:32.253571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bf7b4f-d031-4dbb-b071-653330e479d9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:32.253845 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bf7b4f-d031-4dbb-b071-653330e479d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:32.254113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bf7b4f-d031-4dbb-b071-653330e479d9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:32.254343 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46fb6c5-7b22-42f4-a97b-9d598590e3b3 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:32.262356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 257/502] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:27:32 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:32.263158 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46fb6c5-7b22-42f4-a97b-9d598590e3b3 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:32.264957 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bf7b4f-d031-4dbb-b071-653330e479d9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:32.267223 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bf7b4f-d031-4dbb-b071-653330e479d9 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:32.282232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-d2bf7b4f-d031-4dbb-b071-653330e479d9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:32.282232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:32.282232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:32.282232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:32.282232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:32.282232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:32.282232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:32.282232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:32.282232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:32.282232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:32.282232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:32.282232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:32.282232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:32.282232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:32.282232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:32.282232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:32.283234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:32.283234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:32.283234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:32.283234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:32.283234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:32.283234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:32.283234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:32.283234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:32.283234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:32.283234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:32.283234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:32.283234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:32.283234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:32.283234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:32.283234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:32.283234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:32.283234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:32.289383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:32.289383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:32.289383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:32.289383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 246/503] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:32.289383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bf7b4f-d031-4dbb-b071-653330e479d9 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:32.314012 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:32.362854 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:32.367853 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46fb6c5-7b22-42f4-a97b-9d598590e3b3 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:32.387032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bf7b4f-d031-4dbb-b071-653330e479d9 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:32.401268 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c77775e-cccd-4a8e-8f31-412a0f75d9b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:32.401909 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c77775e-cccd-4a8e-8f31-412a0f75d9b2 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:32.402194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c77775e-cccd-4a8e-8f31-412a0f75d9b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:32.402476 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c77775e-cccd-4a8e-8f31-412a0f75d9b2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:32.420899 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2ae04a-53bb-4285-aa70-a8d32611f195 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:32.421528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2ae04a-53bb-4285-aa70-a8d32611f195 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:32.421810 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2ae04a-53bb-4285-aa70-a8d32611f195 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:32.422084 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2ae04a-53bb-4285-aa70-a8d32611f195 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:32.429640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2ae04a-53bb-4285-aa70-a8d32611f195 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:32.431122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2ae04a-53bb-4285-aa70-a8d32611f195 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:32.442453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-cb2ae04a-53bb-4285-aa70-a8d32611f195 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 16 03:27:32.442453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:32.442453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:32.442453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:32.442453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:32.442453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:32.442453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:32.442453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:32.442453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:32.442453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:32.442453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:32.442453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:32.442453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:27:32.442453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:27:32.442453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:27:32.442453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:27:32.443382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:32.443382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:32.443382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:27:32.443382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:32.443382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:27:32.443382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:32.443382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:32.443382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:32.443382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:32.443382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:32.443382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:32.443382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:32.443382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:32.443382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:32.443382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:32.443382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:32.443382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:32.443382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:32.444263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:32.444263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:32.444263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:27:32.444263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:27:32.444263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:27:32.444263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:27:32.444263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:27:32.444263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:27:32.444263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 16 03:27:32.444263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:32.444263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 247/504] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:32 2022] GET /identity/v3/users/glance => generated 84 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:32.444965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d84f167-1b10-487c-9e1f-8265b39a6efd None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:32.444965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d84f167-1b10-487c-9e1f-8265b39a6efd None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:32.445292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d84f167-1b10-487c-9e1f-8265b39a6efd None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:32.446432 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 248/505] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:32 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:32.447276 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d84f167-1b10-487c-9e1f-8265b39a6efd None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:32.473928 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c77775e-cccd-4a8e-8f31-412a0f75d9b2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:32.475365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c77775e-cccd-4a8e-8f31-412a0f75d9b2 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:32.520074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:32.548793 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d84f167-1b10-487c-9e1f-8265b39a6efd None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:32.573423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 258/506] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:27:32.579302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c77775e-cccd-4a8e-8f31-412a0f75d9b2 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:32.581949 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080c0b50-a37c-4315-acf7-40ba87c53457 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:32.582587 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080c0b50-a37c-4315-acf7-40ba87c53457 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:32.582869 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080c0b50-a37c-4315-acf7-40ba87c53457 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:32.583189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080c0b50-a37c-4315-acf7-40ba87c53457 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:32.585043 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:32.593968 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080c0b50-a37c-4315-acf7-40ba87c53457 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:32.595642 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080c0b50-a37c-4315-acf7-40ba87c53457 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:32.610480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-080c0b50-a37c-4315-acf7-40ba87c53457 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:32.610480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:32.610480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:32.610480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:32.610480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:32.610480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:32.610480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:32.610480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:32.610480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:32.610480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:32.610480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:32.610480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:32.610480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:32.610480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:32.610480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:32.610480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:32.611651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:32.611651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:32.611651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:32.611651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:32.611651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:32.611651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:32.611651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:32.611651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:32.611651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:32.611651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:32.611651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:32.611651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:32.611651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:32.611651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:32.611651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:32.611651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:32.611651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:32.613008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:32.613008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:32.613008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:32.613008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 249/507] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:32.613008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080c0b50-a37c-4315-acf7-40ba87c53457 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:32.679495 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c77775e-cccd-4a8e-8f31-412a0f75d9b2 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:32.709238 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080c0b50-a37c-4315-acf7-40ba87c53457 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:32.712448 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4fcb78-c6f2-4e5b-8f77-c236829656a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:32.713046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4fcb78-c6f2-4e5b-8f77-c236829656a7 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:32.713326 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4fcb78-c6f2-4e5b-8f77-c236829656a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:32.713596 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4fcb78-c6f2-4e5b-8f77-c236829656a7 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:32.714222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4fcb78-c6f2-4e5b-8f77-c236829656a7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:32.714523 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4fcb78-c6f2-4e5b-8f77-c236829656a7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:32.715950 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4fcb78-c6f2-4e5b-8f77-c236829656a7 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:32.727346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 259/508] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:32.728424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9c8671-5548-4572-a638-3071e93f687b None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:32.728714 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9c8671-5548-4572-a638-3071e93f687b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:32.728990 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9c8671-5548-4572-a638-3071e93f687b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:32.742872 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:32.743574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a2056c-6ec6-42c1-b959-5d9490d60c9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:32.744177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a2056c-6ec6-42c1-b959-5d9490d60c9f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:32.744455 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a2056c-6ec6-42c1-b959-5d9490d60c9f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:32.744752 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a2056c-6ec6-42c1-b959-5d9490d60c9f None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:32.745467 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a2056c-6ec6-42c1-b959-5d9490d60c9f None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:32.745766 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a2056c-6ec6-42c1-b959-5d9490d60c9f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:32.747268 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a2056c-6ec6-42c1-b959-5d9490d60c9f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:32.748394 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-fb9c8671-5548-4572-a638-3071e93f687b None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:32.749574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-fb9c8671-5548-4572-a638-3071e93f687b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:32.764663 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 250/509] 10.222.0.49 () {56 vars in 1137 bytes} [Fri Sep 16 03:27:32 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:32.765456 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a2056c-6ec6-42c1-b959-5d9490d60c9f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:32.813732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:32.820817 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 260/510] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:32.821639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9c8671-5548-4572-a638-3071e93f687b None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:32.873390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a2056c-6ec6-42c1-b959-5d9490d60c9f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:32.914061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afa1a7e-532e-4040-92c4-7aea0b96236e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:32.914682 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afa1a7e-532e-4040-92c4-7aea0b96236e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:32.914983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afa1a7e-532e-4040-92c4-7aea0b96236e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:32.915256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afa1a7e-532e-4040-92c4-7aea0b96236e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:32.926540 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9c8671-5548-4572-a638-3071e93f687b None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:32.929561 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afa1a7e-532e-4040-92c4-7aea0b96236e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:32.931102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afa1a7e-532e-4040-92c4-7aea0b96236e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:32.947187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-6afa1a7e-532e-4040-92c4-7aea0b96236e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:32.947187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:32.947187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:32.947187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:32.947187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:32.947187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:32.947187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:32.947187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:32.947187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:32.947187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:32.947187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:32.947187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:32.947187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:32.947187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:32.947187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:32.947187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:32.948119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:32.948119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:32.948119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:32.948119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:32.948119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:32.948119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:32.948119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:32.948119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:32.948119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:32.948119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:32.948119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:32.948119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:32.948119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:32.948119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:32.948119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:32.948119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:32.948119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:32.948978 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:32.948978 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:32.948978 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:32.948978 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 251/511] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:32.949203 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afa1a7e-532e-4040-92c4-7aea0b96236e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:32.967239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8691d252-c6d6-42d2-b44a-0f12240b5263 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:32.967886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8691d252-c6d6-42d2-b44a-0f12240b5263 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:32.968163 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8691d252-c6d6-42d2-b44a-0f12240b5263 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:32.968438 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8691d252-c6d6-42d2-b44a-0f12240b5263 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:32.969073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8691d252-c6d6-42d2-b44a-0f12240b5263 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:32.969398 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8691d252-c6d6-42d2-b44a-0f12240b5263 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:32.970772 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8691d252-c6d6-42d2-b44a-0f12240b5263 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:32.973551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:32.990892 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 261/512] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:32.990892 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8691d252-c6d6-42d2-b44a-0f12240b5263 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:33.056641 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:33.062362 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afa1a7e-532e-4040-92c4-7aea0b96236e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:33.089704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8691d252-c6d6-42d2-b44a-0f12240b5263 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:33.098337 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac337651-52a0-48b7-b0cf-05617b52e716 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:33.099106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac337651-52a0-48b7-b0cf-05617b52e716 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:33.099384 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac337651-52a0-48b7-b0cf-05617b52e716 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:33.099663 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac337651-52a0-48b7-b0cf-05617b52e716 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:33.106772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac337651-52a0-48b7-b0cf-05617b52e716 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:33.108235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac337651-52a0-48b7-b0cf-05617b52e716 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:33.119644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-ac337651-52a0-48b7-b0cf-05617b52e716 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 16 03:27:33.119644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:33.119644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:33.119644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:33.119644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:33.119644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:33.119644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:33.119644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:33.119644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:33.119644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:33.119644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:33.119644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:33.119644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:27:33.119644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:27:33.119644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:27:33.119644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:27:33.120743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:33.120743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:33.120743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:27:33.120743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:33.120743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:27:33.120743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:33.120743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:33.120743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:33.120743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:33.120743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:33.120743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:33.120743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:33.120743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:33.120743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:33.120743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:33.120743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:33.120743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:33.120743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:33.122106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:33.122106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:33.122106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:27:33.122106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:27:33.122106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:27:33.122106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:27:33.122106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:27:33.122106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:27:33.122106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 16 03:27:33.122106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:33.122106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 252/513] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:32 2022] GET /identity/v3/users/cinder => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:33.122106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac337651-52a0-48b7-b0cf-05617b52e716 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:33.126729 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25080dc1-1c7e-427e-890b-f362b44e9682 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:33.127445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25080dc1-1c7e-427e-890b-f362b44e9682 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:33.127731 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25080dc1-1c7e-427e-890b-f362b44e9682 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:33.128006 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25080dc1-1c7e-427e-890b-f362b44e9682 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:33.190625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:33.196883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25080dc1-1c7e-427e-890b-f362b44e9682 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:33.198365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25080dc1-1c7e-427e-890b-f362b44e9682 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:33.219905 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac337651-52a0-48b7-b0cf-05617b52e716 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:33.253379 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70967d6e-4390-4246-8880-fa972eeadc7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:33.253912 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70967d6e-4390-4246-8880-fa972eeadc7b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:33.254219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70967d6e-4390-4246-8880-fa972eeadc7b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:33.254495 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70967d6e-4390-4246-8880-fa972eeadc7b None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:33.255165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70967d6e-4390-4246-8880-fa972eeadc7b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:33.255467 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70967d6e-4390-4246-8880-fa972eeadc7b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:33.256816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70967d6e-4390-4246-8880-fa972eeadc7b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:33.269679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 253/514] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:33.270526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70967d6e-4390-4246-8880-fa972eeadc7b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:33.289500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:33.297923 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 262/515] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:27:33.298756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25080dc1-1c7e-427e-890b-f362b44e9682 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:33.370175 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70967d6e-4390-4246-8880-fa972eeadc7b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:33.392521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:33.398113 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25080dc1-1c7e-427e-890b-f362b44e9682 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:33.409946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f7806f-ebcb-4e20-8b75-f7dcc0caf9e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:33.409946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f7806f-ebcb-4e20-8b75-f7dcc0caf9e4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:33.409946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f7806f-ebcb-4e20-8b75-f7dcc0caf9e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:33.409946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f7806f-ebcb-4e20-8b75-f7dcc0caf9e4 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:33.412966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f7806f-ebcb-4e20-8b75-f7dcc0caf9e4 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:33.414408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f7806f-ebcb-4e20-8b75-f7dcc0caf9e4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:33.426056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-a8f7806f-ebcb-4e20-8b75-f7dcc0caf9e4 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 16 03:27:33.426056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:33.426056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:33.426056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:33.426056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:33.426056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:33.426056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:33.426056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:33.426056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:33.426056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:33.426056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:33.426056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:33.426056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:27:33.426056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:27:33.426056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:27:33.426056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:27:33.427106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:33.427106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:33.427106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:27:33.427106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:33.427106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:27:33.427106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:33.427106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:33.427106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:33.427106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:33.427106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:33.427106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:33.427106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:33.427106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:33.427106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:33.427106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:33.427106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:33.427106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:33.427106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:33.433051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:33.433051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:33.433051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:27:33.433051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:27:33.433051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:27:33.433051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:27:33.433051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:27:33.433051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:27:33.433051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 16 03:27:33.433051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:33.433051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 254/516] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:27:33 2022] GET /identity/v3/users/neutron => generated 85 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:33.433051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f7806f-ebcb-4e20-8b75-f7dcc0caf9e4 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:33.435376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859439d9-f37b-4092-83ae-187abaf9a3ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:33.435376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859439d9-f37b-4092-83ae-187abaf9a3ca None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:33.435600 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859439d9-f37b-4092-83ae-187abaf9a3ca None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:33.435793 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859439d9-f37b-4092-83ae-187abaf9a3ca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:33.446464 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859439d9-f37b-4092-83ae-187abaf9a3ca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:33.447937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859439d9-f37b-4092-83ae-187abaf9a3ca None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:33.462419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-859439d9-f37b-4092-83ae-187abaf9a3ca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:33.462419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:33.462419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:33.462419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:33.462419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:33.462419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:33.462419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:33.462419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:33.462419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:33.462419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:33.462419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:33.462419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:33.462419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:33.462419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:33.462419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:33.462419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:33.463513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:33.463513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:33.463513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:33.463513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:33.463513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:33.463513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:33.463513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:33.463513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:33.463513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:33.463513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:33.463513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:33.463513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:33.463513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:33.463513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:33.463513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:33.463513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:33.463513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:33.464377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:33.464377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:33.464377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:33.464377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 263/517] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:33.464377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859439d9-f37b-4092-83ae-187abaf9a3ca None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:33.533524 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:33.556996 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f7806f-ebcb-4e20-8b75-f7dcc0caf9e4 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:33.566888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859439d9-f37b-4092-83ae-187abaf9a3ca None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:33.591720 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e459e70f-8240-40a6-a161-fbaa5df46b8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:33.592326 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e459e70f-8240-40a6-a161-fbaa5df46b8c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:33.592607 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e459e70f-8240-40a6-a161-fbaa5df46b8c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:33.592879 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e459e70f-8240-40a6-a161-fbaa5df46b8c None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:33.593608 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e459e70f-8240-40a6-a161-fbaa5df46b8c None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:33.593910 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e459e70f-8240-40a6-a161-fbaa5df46b8c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:33.595361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e459e70f-8240-40a6-a161-fbaa5df46b8c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:33.600583 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c35cc31-e04d-48c5-ba89-6ac4db29689b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:33.601202 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c35cc31-e04d-48c5-ba89-6ac4db29689b None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:33.601482 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c35cc31-e04d-48c5-ba89-6ac4db29689b None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:33.601755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c35cc31-e04d-48c5-ba89-6ac4db29689b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:33.609141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c35cc31-e04d-48c5-ba89-6ac4db29689b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:33.610581 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c35cc31-e04d-48c5-ba89-6ac4db29689b None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:33.612189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 255/518] 10.222.0.49 () {56 vars in 1137 bytes} [Fri Sep 16 03:27:33 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:33.612982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e459e70f-8240-40a6-a161-fbaa5df46b8c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:33.622684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-0c35cc31-e04d-48c5-ba89-6ac4db29689b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:33.622684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:33.622684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:33.622684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:33.622684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:33.622684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:33.622684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:33.622684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:33.622684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:33.622684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:33.622684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:33.622684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:33.622684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:27:33.622684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:27:33.622684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:27:33.623625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:27:33.623625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:33.623625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:33.623625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:33.623625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:33.623625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:33.623625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:33.623625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:33.623625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:33.623625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:33.623625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:33.623625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:33.623625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:33.623625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:33.623625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:33.623625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:33.623625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:33.624465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:27:33.624465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:27:33.624465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:27:33.624465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:27:33.624465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:27:33.624465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:27:33.624465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:33.624465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:33.624465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 264/519] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:33.625058 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c35cc31-e04d-48c5-ba89-6ac4db29689b None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:33.639246 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:33.716003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e459e70f-8240-40a6-a161-fbaa5df46b8c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:33.726744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c35cc31-e04d-48c5-ba89-6ac4db29689b None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:33.749929 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d434384-954e-4d60-b63c-2680e93b7fcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:33.750604 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d434384-954e-4d60-b63c-2680e93b7fcd None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:33.750894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d434384-954e-4d60-b63c-2680e93b7fcd None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:33.751283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d434384-954e-4d60-b63c-2680e93b7fcd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:33.753159 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:33.759120 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c89c31-328d-4a81-9bd1-688365bbd58e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:33.776045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c89c31-328d-4a81-9bd1-688365bbd58e None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:33.776045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c89c31-328d-4a81-9bd1-688365bbd58e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:33.776045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c89c31-328d-4a81-9bd1-688365bbd58e None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:33.776045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c89c31-328d-4a81-9bd1-688365bbd58e None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:33.776045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c89c31-328d-4a81-9bd1-688365bbd58e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:33.776411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d434384-954e-4d60-b63c-2680e93b7fcd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:33.776488 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c89c31-328d-4a81-9bd1-688365bbd58e None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:33.776566 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d434384-954e-4d60-b63c-2680e93b7fcd None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:33.777242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-4d434384-954e-4d60-b63c-2680e93b7fcd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:33.777242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:33.777242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:33.777242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:33.777242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:33.777242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:33.777242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:33.777242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:33.777242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:33.777242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:33.777242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:33.777242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:33.777242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:33.777242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:33.777242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:33.777242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:33.778212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:33.778212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:33.778212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:33.778212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:33.778212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:33.778212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:33.778212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:33.778212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:33.778212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:33.778212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:33.778212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:33.778212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:33.778212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:33.778212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:33.778212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:33.778212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:33.778212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:33.779117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:33.779117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:33.779117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:33.779117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 256/520] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:33.779373 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d434384-954e-4d60-b63c-2680e93b7fcd None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:33.804402 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 265/521] 10.222.0.49 () {56 vars in 1139 bytes} [Fri Sep 16 03:27:33 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:33.804402 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c89c31-328d-4a81-9bd1-688365bbd58e None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:33.873676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:33.892703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d434384-954e-4d60-b63c-2680e93b7fcd None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:33.907580 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c89c31-328d-4a81-9bd1-688365bbd58e None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:33.927166 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022f5dd5-ae68-4f36-b563-4ba948dcabe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:33.927775 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022f5dd5-ae68-4f36-b563-4ba948dcabe6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:33.928065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022f5dd5-ae68-4f36-b563-4ba948dcabe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:33.928339 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022f5dd5-ae68-4f36-b563-4ba948dcabe6 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:33.928980 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022f5dd5-ae68-4f36-b563-4ba948dcabe6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:33.929285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022f5dd5-ae68-4f36-b563-4ba948dcabe6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:33.930652 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022f5dd5-ae68-4f36-b563-4ba948dcabe6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:33.941858 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dda923a-993b-4c96-bb58-493a82c60659 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:33.942370 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 257/522] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:33.946594 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dda923a-993b-4c96-bb58-493a82c60659 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:33.946594 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dda923a-993b-4c96-bb58-493a82c60659 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:33.946594 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dda923a-993b-4c96-bb58-493a82c60659 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:33.946798 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022f5dd5-ae68-4f36-b563-4ba948dcabe6 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:33.994659 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:34.012890 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dda923a-993b-4c96-bb58-493a82c60659 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:34.014370 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dda923a-993b-4c96-bb58-493a82c60659 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:34.046661 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022f5dd5-ae68-4f36-b563-4ba948dcabe6 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:34.077464 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:34.081321 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f5c22b-6dd9-443c-8388-6a2cf5694352 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:34.081928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f5c22b-6dd9-443c-8388-6a2cf5694352 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:34.082209 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f5c22b-6dd9-443c-8388-6a2cf5694352 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:34.082492 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f5c22b-6dd9-443c-8388-6a2cf5694352 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:34.083211 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f5c22b-6dd9-443c-8388-6a2cf5694352 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:34.083518 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f5c22b-6dd9-443c-8388-6a2cf5694352 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:34.084932 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f5c22b-6dd9-443c-8388-6a2cf5694352 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:34.102857 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 258/523] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:27:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:34.103686 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f5c22b-6dd9-443c-8388-6a2cf5694352 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:34.117881 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 266/524] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:27:34.118918 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dda923a-993b-4c96-bb58-493a82c60659 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:34.208361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f5c22b-6dd9-443c-8388-6a2cf5694352 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:34.215516 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:34.223393 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dda923a-993b-4c96-bb58-493a82c60659 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:34.242579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284ef39d-8a30-45a1-9adb-92193555d11a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:34.243215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284ef39d-8a30-45a1-9adb-92193555d11a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:34.243496 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284ef39d-8a30-45a1-9adb-92193555d11a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:34.243767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284ef39d-8a30-45a1-9adb-92193555d11a None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:34.244406 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284ef39d-8a30-45a1-9adb-92193555d11a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:34.244707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284ef39d-8a30-45a1-9adb-92193555d11a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:34.246064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284ef39d-8a30-45a1-9adb-92193555d11a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:34.256797 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758f1e2d-ee12-44af-b894-f133e18ba804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:34.257395 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758f1e2d-ee12-44af-b894-f133e18ba804 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:34.257676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758f1e2d-ee12-44af-b894-f133e18ba804 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:34.257949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758f1e2d-ee12-44af-b894-f133e18ba804 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:34.258237 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 259/525] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:34.259040 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284ef39d-8a30-45a1-9adb-92193555d11a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:34.295760 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:34.330062 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758f1e2d-ee12-44af-b894-f133e18ba804 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:34.331510 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758f1e2d-ee12-44af-b894-f133e18ba804 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:34.363086 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284ef39d-8a30-45a1-9adb-92193555d11a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:34.397150 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2b6ec7-9e44-453e-96b3-2dd79e1272b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:34.397743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2b6ec7-9e44-453e-96b3-2dd79e1272b6 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:34.398017 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2b6ec7-9e44-453e-96b3-2dd79e1272b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:34.398287 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2b6ec7-9e44-453e-96b3-2dd79e1272b6 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:34.398940 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2b6ec7-9e44-453e-96b3-2dd79e1272b6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:34.400348 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2b6ec7-9e44-453e-96b3-2dd79e1272b6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:34.427657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 267/526] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:27:34.428483 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758f1e2d-ee12-44af-b894-f133e18ba804 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:34.479850 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 260/527] 10.222.0.49 () {60 vars in 1132 bytes} [Fri Sep 16 03:27:34 2022] POST /identity/v3/users => generated 247 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:27:34.480679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2b6ec7-9e44-453e-96b3-2dd79e1272b6 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:34.493078 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:34.508342 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:34.536516 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758f1e2d-ee12-44af-b894-f133e18ba804 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:34.576661 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493b25f6-e83f-4d4f-8970-a2f20908675c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:34.577288 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493b25f6-e83f-4d4f-8970-a2f20908675c None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:34.577569 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493b25f6-e83f-4d4f-8970-a2f20908675c None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:34.577842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493b25f6-e83f-4d4f-8970-a2f20908675c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:34.579126 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493b25f6-e83f-4d4f-8970-a2f20908675c None admin] RBAC: Adding query filter params (role.id=974d839067e9411aa4cb850cee33f6ae, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=ded1bcd10e62499a9aaa0ece5aa55c80) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:34.579829 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493b25f6-e83f-4d4f-8970-a2f20908675c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:34.581231 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493b25f6-e83f-4d4f-8970-a2f20908675c None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:34.587480 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 268/528] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:27:34 2022] GET /identity/v3/role_assignments?user.id=ded1bcd10e62499a9aaa0ece5aa55c80&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=974d839067e9411aa4cb850cee33f6ae => generated 259 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:34.592961 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2b6ec7-9e44-453e-96b3-2dd79e1272b6 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:34.599873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493b25f6-e83f-4d4f-8970-a2f20908675c None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:34.648836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46edb9de-171b-4bfe-906d-cc3e07ab639f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:34.650735 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46edb9de-171b-4bfe-906d-cc3e07ab639f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:34.651370 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46edb9de-171b-4bfe-906d-cc3e07ab639f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:34.651849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46edb9de-171b-4bfe-906d-cc3e07ab639f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:34.668709 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46edb9de-171b-4bfe-906d-cc3e07ab639f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:34.670990 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46edb9de-171b-4bfe-906d-cc3e07ab639f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:34.693784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-46edb9de-171b-4bfe-906d-cc3e07ab639f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:34.693784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:34.693784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:34.693784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:34.693784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:34.693784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:34.693784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:34.693784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:34.693784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:34.693784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:34.693784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:34.693784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:34.693784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:34.693784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:34.693784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:34.693784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:34.695056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:34.695056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:34.695056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:34.695056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:34.695056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:34.695056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:34.695056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:34.695056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:34.695056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:34.695056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:34.695056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:34.695056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:34.695056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:34.695056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:34.695056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:34.695056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:34.695056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:34.696151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:34.696151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:34.696151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:34.696151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 261/529] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:34.696434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46edb9de-171b-4bfe-906d-cc3e07ab639f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:34.742465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493b25f6-e83f-4d4f-8970-a2f20908675c None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:34.777269 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:34.783877 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0096852-8047-402d-aa35-562f4289fea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:34.784764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0096852-8047-402d-aa35-562f4289fea9 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:34.784883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0096852-8047-402d-aa35-562f4289fea9 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:34.787158 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0096852-8047-402d-aa35-562f4289fea9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:34.793091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0096852-8047-402d-aa35-562f4289fea9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:34.794637 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0096852-8047-402d-aa35-562f4289fea9 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:34.799408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:34.807374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-d0096852-8047-402d-aa35-562f4289fea9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:34.807374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:34.807374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:34.807374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:34.807374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:34.807374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:34.807374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:34.807374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:34.807374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:34.807374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:34.807374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:34.807374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:34.807374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:27:34.807374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:27:34.807374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:27:34.808449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:27:34.808449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:34.808449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:34.808449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:34.808449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:34.808449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:34.808449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:34.808449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:34.808449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:34.808449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:34.808449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:34.808449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:34.808449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:34.808449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:34.808449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:34.808449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:34.808449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:34.812109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:27:34.812109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:27:34.812109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:27:34.812109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:27:34.812109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:27:34.812109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:27:34.812109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:34.812109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:34.812109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 269/530] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:34.812109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0096852-8047-402d-aa35-562f4289fea9 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:34.831488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46edb9de-171b-4bfe-906d-cc3e07ab639f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:34.877048 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7f49b6-fb74-4e6d-8338-8118b391177e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:34.877720 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7f49b6-fb74-4e6d-8338-8118b391177e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:34.877998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7f49b6-fb74-4e6d-8338-8118b391177e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:34.878277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7f49b6-fb74-4e6d-8338-8118b391177e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:34.892348 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7f49b6-fb74-4e6d-8338-8118b391177e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:34.893400 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7f49b6-fb74-4e6d-8338-8118b391177e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:34.910434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-af7f49b6-fb74-4e6d-8338-8118b391177e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:34.910434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:34.910434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:34.910434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:34.910434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:34.910434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:34.910434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:34.910434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:34.910434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:34.910434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:34.910434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:34.910434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:34.910434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:34.910434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:34.910434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:34.910434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:34.911429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:34.911429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:34.911429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:34.911429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:34.911429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:34.911429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:34.911429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:34.911429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:34.911429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:34.911429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:34.911429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:34.911429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:34.911429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:34.911429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:34.911429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:34.911429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:34.911429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:34.912273 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:34.912273 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:34.912273 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:34.912273 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 262/531] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:34.912907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7f49b6-fb74-4e6d-8338-8118b391177e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:34.944124 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0096852-8047-402d-aa35-562f4289fea9 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:34.981857 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15addd6-38d4-4c2f-ad1b-8a476cd53c9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:34.982878 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15addd6-38d4-4c2f-ad1b-8a476cd53c9b None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:34.983478 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15addd6-38d4-4c2f-ad1b-8a476cd53c9b None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:34.983987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15addd6-38d4-4c2f-ad1b-8a476cd53c9b None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:34.985020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15addd6-38d4-4c2f-ad1b-8a476cd53c9b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:34.985547 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15addd6-38d4-4c2f-ad1b-8a476cd53c9b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:34.987293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15addd6-38d4-4c2f-ad1b-8a476cd53c9b None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:35.001788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 270/532] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:35.002957 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15addd6-38d4-4c2f-ad1b-8a476cd53c9b None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:35.031338 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7f49b6-fb74-4e6d-8338-8118b391177e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:35.056908 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:35.061730 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:35.072215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd52b6c3-83c5-4445-9027-b45ef5f2cef4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:35.072928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd52b6c3-83c5-4445-9027-b45ef5f2cef4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:35.074676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd52b6c3-83c5-4445-9027-b45ef5f2cef4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:35.074676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd52b6c3-83c5-4445-9027-b45ef5f2cef4 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:35.074676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd52b6c3-83c5-4445-9027-b45ef5f2cef4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:35.074676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd52b6c3-83c5-4445-9027-b45ef5f2cef4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:35.076168 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd52b6c3-83c5-4445-9027-b45ef5f2cef4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:35.095033 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 263/533] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:27:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:35.095033 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd52b6c3-83c5-4445-9027-b45ef5f2cef4 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:35.127164 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15addd6-38d4-4c2f-ad1b-8a476cd53c9b None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:35.166063 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7491b2e-24cb-44b0-a9eb-4920ab20db72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:35.166736 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7491b2e-24cb-44b0-a9eb-4920ab20db72 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:35.167047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7491b2e-24cb-44b0-a9eb-4920ab20db72 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:35.167322 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7491b2e-24cb-44b0-a9eb-4920ab20db72 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:35.168152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7491b2e-24cb-44b0-a9eb-4920ab20db72 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:35.168915 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7491b2e-24cb-44b0-a9eb-4920ab20db72 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:35.170695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7491b2e-24cb-44b0-a9eb-4920ab20db72 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:35.184852 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 271/534] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:35.185819 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7491b2e-24cb-44b0-a9eb-4920ab20db72 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:35.213126 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd52b6c3-83c5-4445-9027-b45ef5f2cef4 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:35.257116 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db62dd1-2dd2-4072-b379-864a16429f15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:35.257751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db62dd1-2dd2-4072-b379-864a16429f15 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:35.258036 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db62dd1-2dd2-4072-b379-864a16429f15 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:35.258855 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db62dd1-2dd2-4072-b379-864a16429f15 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:35.268693 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db62dd1-2dd2-4072-b379-864a16429f15 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:35.270851 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db62dd1-2dd2-4072-b379-864a16429f15 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:35.287318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-2db62dd1-2dd2-4072-b379-864a16429f15 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:35.287318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:35.287318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:35.287318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:35.287318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:35.287318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:35.287318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:35.287318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:35.287318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:35.287318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:35.287318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:35.287318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:35.287318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:27:35.287318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:27:35.287318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:27:35.288277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:27:35.288277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:35.288277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:35.288277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:35.288277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:35.288277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:35.288277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:35.288277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:35.288277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:35.288277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:35.288277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:35.288277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:35.288277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:35.288277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:35.288277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:35.288277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:35.288277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:35.289113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:27:35.289113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:27:35.289113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:27:35.289113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:27:35.289113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:27:35.289113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:27:35.289113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:35.289113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:35.289517 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 264/535] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:35.294931 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db62dd1-2dd2-4072-b379-864a16429f15 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:35.325551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7491b2e-24cb-44b0-a9eb-4920ab20db72 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:35.337880 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:35.359085 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:35.364426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6aad4b6-e719-41bb-8e3e-24e72efc7f1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:35.365157 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6aad4b6-e719-41bb-8e3e-24e72efc7f1d None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:35.365533 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6aad4b6-e719-41bb-8e3e-24e72efc7f1d None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:35.365920 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6aad4b6-e719-41bb-8e3e-24e72efc7f1d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:35.366773 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6aad4b6-e719-41bb-8e3e-24e72efc7f1d None admin] RBAC: Adding query filter params (role.id=974d839067e9411aa4cb850cee33f6ae, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=05ceb9fc31c345ccabaee8f30a3982a8) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:35.367221 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6aad4b6-e719-41bb-8e3e-24e72efc7f1d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:35.369500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6aad4b6-e719-41bb-8e3e-24e72efc7f1d None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:35.383272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 272/536] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:27:35 2022] GET /identity/v3/role_assignments?user.id=05ceb9fc31c345ccabaee8f30a3982a8&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=974d839067e9411aa4cb850cee33f6ae => generated 259 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:35.384511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6aad4b6-e719-41bb-8e3e-24e72efc7f1d None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:35.425537 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db62dd1-2dd2-4072-b379-864a16429f15 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:35.469338 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003084f2-62cb-40c7-9f12-f6c81f362359 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:35.470439 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003084f2-62cb-40c7-9f12-f6c81f362359 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:35.470944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003084f2-62cb-40c7-9f12-f6c81f362359 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:35.471446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003084f2-62cb-40c7-9f12-f6c81f362359 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:35.483880 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003084f2-62cb-40c7-9f12-f6c81f362359 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:35.486283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003084f2-62cb-40c7-9f12-f6c81f362359 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:35.502881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-003084f2-62cb-40c7-9f12-f6c81f362359 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:35.502881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:35.502881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:35.502881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:35.502881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:35.502881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:35.502881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:35.502881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:35.502881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:35.502881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:35.502881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:35.502881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:35.502881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:27:35.502881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:27:35.502881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:27:35.503831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:27:35.503831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:35.503831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:35.503831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:35.503831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:35.503831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:35.503831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:35.503831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:35.503831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:35.503831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:35.503831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:35.503831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:35.503831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:35.503831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:35.503831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:35.503831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:35.503831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:35.504686 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:27:35.504686 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:27:35.504686 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:27:35.504686 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:27:35.504686 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:27:35.504686 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:27:35.504686 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:35.504686 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:35.514975 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 265/537] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:35.537862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003084f2-62cb-40c7-9f12-f6c81f362359 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:35.576577 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6aad4b6-e719-41bb-8e3e-24e72efc7f1d None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:35.647640 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47e7628-f84c-494b-b446-1bdfe90e7609 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:35.648240 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47e7628-f84c-494b-b446-1bdfe90e7609 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:35.648639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47e7628-f84c-494b-b446-1bdfe90e7609 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:35.649067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47e7628-f84c-494b-b446-1bdfe90e7609 None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:35.649792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47e7628-f84c-494b-b446-1bdfe90e7609 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:35.650199 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47e7628-f84c-494b-b446-1bdfe90e7609 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:35.651967 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47e7628-f84c-494b-b446-1bdfe90e7609 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:35.675715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 273/538] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:27:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:35.682066 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47e7628-f84c-494b-b446-1bdfe90e7609 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:35.698241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003084f2-62cb-40c7-9f12-f6c81f362359 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:35.706076 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:35.724433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:35.761045 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faea7c25-5083-4f25-8e1f-c9bb16def445 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:35.762180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faea7c25-5083-4f25-8e1f-c9bb16def445 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:35.762180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faea7c25-5083-4f25-8e1f-c9bb16def445 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:35.762505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faea7c25-5083-4f25-8e1f-c9bb16def445 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:35.763331 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faea7c25-5083-4f25-8e1f-c9bb16def445 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:35.763638 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faea7c25-5083-4f25-8e1f-c9bb16def445 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:35.765057 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faea7c25-5083-4f25-8e1f-c9bb16def445 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:35.793996 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 266/539] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:27:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:35.807048 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47e7628-f84c-494b-b446-1bdfe90e7609 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:35.814671 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faea7c25-5083-4f25-8e1f-c9bb16def445 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:35.857408 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f9a877-5c1b-4d05-b56f-cb4e3d4a483d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:35.858028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f9a877-5c1b-4d05-b56f-cb4e3d4a483d None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:35.858307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f9a877-5c1b-4d05-b56f-cb4e3d4a483d None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:35.858578 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f9a877-5c1b-4d05-b56f-cb4e3d4a483d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:35.859342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f9a877-5c1b-4d05-b56f-cb4e3d4a483d None admin] RBAC: Adding query filter params (role.id=974d839067e9411aa4cb850cee33f6ae, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=e6aeffd924734771bc5bf0d282c9052d) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:35.859652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f9a877-5c1b-4d05-b56f-cb4e3d4a483d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:35.864326 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f9a877-5c1b-4d05-b56f-cb4e3d4a483d None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:35.876208 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 274/540] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:27:35 2022] GET /identity/v3/role_assignments?user.id=e6aeffd924734771bc5bf0d282c9052d&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=974d839067e9411aa4cb850cee33f6ae => generated 259 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:35.983116 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faea7c25-5083-4f25-8e1f-c9bb16def445 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:36.040339 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a88fc6-ece1-4385-a37e-2b51a8a200a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:36.040956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a88fc6-ece1-4385-a37e-2b51a8a200a6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:36.041233 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a88fc6-ece1-4385-a37e-2b51a8a200a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:36.041500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a88fc6-ece1-4385-a37e-2b51a8a200a6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:36.042239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a88fc6-ece1-4385-a37e-2b51a8a200a6 None admin] RBAC: Adding query filter params (role.id=974d839067e9411aa4cb850cee33f6ae, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=f9275eeba471423e9fa2a31a6e08676d) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:36.042543 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a88fc6-ece1-4385-a37e-2b51a8a200a6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:36.044003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a88fc6-ece1-4385-a37e-2b51a8a200a6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:36.055781 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 267/541] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:27:35 2022] GET /identity/v3/role_assignments?user.id=f9275eeba471423e9fa2a31a6e08676d&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=974d839067e9411aa4cb850cee33f6ae => generated 259 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:37.028759 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ed48e8-4da8-4ec8-ae4c-e5998223ba3c None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:37.029276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ed48e8-4da8-4ec8-ae4c-e5998223ba3c None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:37.029352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ed48e8-4da8-4ec8-ae4c-e5998223ba3c None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:37.031292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 275/542] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:37.036855 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f31233-2aec-4eb1-98d3-200583c29bc7 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:37.037333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f31233-2aec-4eb1-98d3-200583c29bc7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:37.037410 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f31233-2aec-4eb1-98d3-200583c29bc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:37.064110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-34f31233-2aec-4eb1-98d3-200583c29bc7 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:37.065351 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-34f31233-2aec-4eb1-98d3-200583c29bc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:37.071464 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69ca346-ff38-4de1-b90b-548b4f81afe1 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:37.072028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69ca346-ff38-4de1-b90b-548b4f81afe1 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:37.072511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69ca346-ff38-4de1-b90b-548b4f81afe1 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:37.074267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 276/543] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:37.080241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bf3af0-383f-4e84-a0c5-744fdda0ef74 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:37.080938 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bf3af0-383f-4e84-a0c5-744fdda0ef74 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:37.081433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bf3af0-383f-4e84-a0c5-744fdda0ef74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:37.099918 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-a4bf3af0-383f-4e84-a0c5-744fdda0ef74 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:37.101432 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:37.101647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a4bf3af0-383f-4e84-a0c5-744fdda0ef74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:37.176069 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 268/544] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:37.213398 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 277/545] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 03:27:37.489349 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f31233-2aec-4eb1-98d3-200583c29bc7 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:37.510182 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bf3af0-383f-4e84-a0c5-744fdda0ef74 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:37.570720 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:37.604371 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f31233-2aec-4eb1-98d3-200583c29bc7 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:37.665030 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696a778b-17ba-47a4-a720-eca75bfa4bed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:37.666017 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696a778b-17ba-47a4-a720-eca75bfa4bed None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:37.666496 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696a778b-17ba-47a4-a720-eca75bfa4bed None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:37.666996 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696a778b-17ba-47a4-a720-eca75bfa4bed None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:37.678252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696a778b-17ba-47a4-a720-eca75bfa4bed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:37.683473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-696a778b-17ba-47a4-a720-eca75bfa4bed None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:37.683473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:37.683473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:37.683473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:37.683473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:37.683473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:37.683473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:37.683473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:37.683473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:37.683473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:37.683473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:37.683473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:37.683473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:37.683473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:37.683473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:37.683473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:37.684335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:37.684335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:37.684335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:37.684335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:37.684335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:37.684335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:37.684335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:37.684335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:37.684335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:37.684335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:37.684335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:37.684335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:37.684335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:37.684335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:37.684335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:37.684335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:37.685137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:37.685137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:37.685137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:37.685137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:37.685137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:37.685137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:37.685137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:37.685137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:37.685137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:37.685137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:37.685137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:37.685137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:37.687369 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 269/546] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:27:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:37.693182 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696a778b-17ba-47a4-a720-eca75bfa4bed None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:37.705844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bf3af0-383f-4e84-a0c5-744fdda0ef74 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:37.731338 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:37.776552 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72642f67-1c01-446f-86ac-c643455b08d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:37.777264 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72642f67-1c01-446f-86ac-c643455b08d7 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:37.777554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72642f67-1c01-446f-86ac-c643455b08d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:37.777832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72642f67-1c01-446f-86ac-c643455b08d7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:37.807939 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72642f67-1c01-446f-86ac-c643455b08d7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:37.811571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696a778b-17ba-47a4-a720-eca75bfa4bed None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:37.813026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-72642f67-1c01-446f-86ac-c643455b08d7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:37.813026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:37.813026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:37.813026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:37.813026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:37.813026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:37.813026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:37.813026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:37.813026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:37.813026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:37.813026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:37.813026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:37.813026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:37.813026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:37.813026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:37.813026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:37.814160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:37.814160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:37.814160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:37.814160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:37.814160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:37.814160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:37.814160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:37.814160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:37.814160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:37.814160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:37.814160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:37.814160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:37.814160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:37.814160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:37.814160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:37.814160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:37.815016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:37.815016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:37.815016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:37.815016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:37.815016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:37.815016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:37.815016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:37.815016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:37.815016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:37.815016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:37.815016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:37.815016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:37.815016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 278/547] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:27:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:37.823886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72642f67-1c01-446f-86ac-c643455b08d7 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:37.857158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d49c3f6-c4fa-4d9a-b925-d2b8f3a296cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:37.858250 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d49c3f6-c4fa-4d9a-b925-d2b8f3a296cf None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:37.858826 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d49c3f6-c4fa-4d9a-b925-d2b8f3a296cf None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:37.859342 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d49c3f6-c4fa-4d9a-b925-d2b8f3a296cf None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:37.860825 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d49c3f6-c4fa-4d9a-b925-d2b8f3a296cf None admin] RBAC: Adding query filter params (name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:37.861411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d49c3f6-c4fa-4d9a-b925-d2b8f3a296cf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:37.865848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d49c3f6-c4fa-4d9a-b925-d2b8f3a296cf None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:37.876197 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 270/548] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:27:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:37.882287 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d49c3f6-c4fa-4d9a-b925-d2b8f3a296cf None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:37.979865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:37.995166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72642f67-1c01-446f-86ac-c643455b08d7 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:38.019633 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:38.037119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdba459f-0a9c-4314-947c-cc25f414d0c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:38.039219 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdba459f-0a9c-4314-947c-cc25f414d0c3 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:38.039889 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdba459f-0a9c-4314-947c-cc25f414d0c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:38.040383 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdba459f-0a9c-4314-947c-cc25f414d0c3 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:38.041354 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdba459f-0a9c-4314-947c-cc25f414d0c3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:38.041929 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdba459f-0a9c-4314-947c-cc25f414d0c3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:38.043817 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdba459f-0a9c-4314-947c-cc25f414d0c3 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:38.056369 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 279/549] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:27:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:38.063843 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdba459f-0a9c-4314-947c-cc25f414d0c3 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:38.081226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d49c3f6-c4fa-4d9a-b925-d2b8f3a296cf None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:38.147620 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79da32be-5057-4242-8d95-a5e030936694 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:38.148173 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79da32be-5057-4242-8d95-a5e030936694 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:38.148449 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79da32be-5057-4242-8d95-a5e030936694 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:38.148712 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79da32be-5057-4242-8d95-a5e030936694 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:38.168209 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79da32be-5057-4242-8d95-a5e030936694 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:38.169673 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79da32be-5057-4242-8d95-a5e030936694 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:38.197794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-79da32be-5057-4242-8d95-a5e030936694 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:38.197794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:38.197794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:38.197794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:38.197794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:38.197794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:38.197794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:38.197794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:38.197794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:38.197794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:38.197794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:38.197794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:38.197794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:38.197794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:38.197794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:38.197794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:38.199063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:38.199063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:38.199063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:38.199063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:38.199063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:38.199063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:38.199063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:38.199063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:38.199063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:38.199063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:38.199063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:38.199063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:38.199063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:38.199063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:38.199063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:38.199063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:38.199063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:38.200062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:38.200062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:38.200062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:38.200062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 271/550] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 317 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:27:38.200814 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de43b8ac-55ee-4a4f-b62d-9c70c05b438d None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:38.200814 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de43b8ac-55ee-4a4f-b62d-9c70c05b438d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:38.201099 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de43b8ac-55ee-4a4f-b62d-9c70c05b438d None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:38.202444 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 272/551] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:38 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:38.206702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de43b8ac-55ee-4a4f-b62d-9c70c05b438d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:38.225512 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdba459f-0a9c-4314-947c-cc25f414d0c3 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:38.272596 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59443a32-75e9-4256-ad9b-56db8e2ee42c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:38.274240 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59443a32-75e9-4256-ad9b-56db8e2ee42c None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:38.275016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59443a32-75e9-4256-ad9b-56db8e2ee42c None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:38.275533 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59443a32-75e9-4256-ad9b-56db8e2ee42c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:38.290865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59443a32-75e9-4256-ad9b-56db8e2ee42c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:38.293303 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59443a32-75e9-4256-ad9b-56db8e2ee42c None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:38.317139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-59443a32-75e9-4256-ad9b-56db8e2ee42c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:38.317139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:38.317139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:38.317139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:38.317139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:38.317139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:38.317139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:38.317139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:38.317139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:38.317139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:38.317139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:38.317139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:38.317139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:38.317139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:38.317139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:38.317139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:38.318122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:38.318122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:38.318122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:38.318122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:38.318122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:38.318122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:38.318122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:38.318122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:38.318122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:38.318122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:38.318122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:38.318122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:38.318122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:38.318122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:38.318122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:38.318122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:38.318122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:38.319489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:38.319489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:38.319489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:38.325591 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 280/552] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:38.325591 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7f2515-3c1e-4388-8ddd-9aaede97522f None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:38.325591 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7f2515-3c1e-4388-8ddd-9aaede97522f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:38.325591 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7f2515-3c1e-4388-8ddd-9aaede97522f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:38.331152 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de43b8ac-55ee-4a4f-b62d-9c70c05b438d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:38.342321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:38.355759 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-5e7f2515-3c1e-4388-8ddd-9aaede97522f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:38.360510 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-5e7f2515-3c1e-4388-8ddd-9aaede97522f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:38.370583 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a5912b-fc3c-495e-9653-5981425af9fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:38.371375 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a5912b-fc3c-495e-9653-5981425af9fb None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:38.371755 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a5912b-fc3c-495e-9653-5981425af9fb None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:38.372128 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a5912b-fc3c-495e-9653-5981425af9fb None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:38.372869 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a5912b-fc3c-495e-9653-5981425af9fb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:38.373268 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a5912b-fc3c-495e-9653-5981425af9fb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:38.374840 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a5912b-fc3c-495e-9653-5981425af9fb None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:38.376768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:38.390429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 273/553] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:38.392270 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a5912b-fc3c-495e-9653-5981425af9fb None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:38.454894 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 281/554] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:38.454894 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7f2515-3c1e-4388-8ddd-9aaede97522f None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:38.560397 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a5912b-fc3c-495e-9653-5981425af9fb None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:38.581391 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7f2515-3c1e-4388-8ddd-9aaede97522f None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:38.619226 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:38.623792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8283ce-ac3a-49f3-b327-18cec376ff01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:38.624410 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8283ce-ac3a-49f3-b327-18cec376ff01 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:38.624687 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8283ce-ac3a-49f3-b327-18cec376ff01 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:38.624959 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8283ce-ac3a-49f3-b327-18cec376ff01 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:38.626155 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8283ce-ac3a-49f3-b327-18cec376ff01 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:38.626564 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8283ce-ac3a-49f3-b327-18cec376ff01 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:38.627925 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8283ce-ac3a-49f3-b327-18cec376ff01 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:38.634589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b944efd9-fef9-4115-af9d-0e53a32b5bae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:38.634589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b944efd9-fef9-4115-af9d-0e53a32b5bae None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:38.634589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b944efd9-fef9-4115-af9d-0e53a32b5bae None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:38.634589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b944efd9-fef9-4115-af9d-0e53a32b5bae None admin] PATH_INFO: `/v3/users/swift` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:38.644467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b944efd9-fef9-4115-af9d-0e53a32b5bae None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:38.645943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b944efd9-fef9-4115-af9d-0e53a32b5bae None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:38.659552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 274/555] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:38.665012 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8283ce-ac3a-49f3-b327-18cec376ff01 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:38.668883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-b944efd9-fef9-4115-af9d-0e53a32b5bae None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 16 03:27:38.668883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:38.668883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:38.668883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:38.668883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:38.668883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:38.668883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:38.668883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:38.668883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:38.668883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:38.668883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:38.668883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:38.668883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:27:38.668883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:27:38.668883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:27:38.668883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:27:38.669856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:38.669856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:38.669856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:27:38.669856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:38.669856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:27:38.669856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:38.669856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:38.669856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:38.669856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:38.669856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:38.669856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:38.669856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:38.669856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:38.669856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:38.669856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:38.669856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:38.669856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:38.669856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:38.671446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:38.671446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:38.671446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:27:38.671446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:27:38.671446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:27:38.671446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:27:38.671446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:27:38.671446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:27:38.671446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 16 03:27:38.671446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:38.671446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 282/556] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:27:38 2022] GET /identity/v3/users/swift => generated 83 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:38.680872 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b944efd9-fef9-4115-af9d-0e53a32b5bae None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:38.745176 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:38.802841 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8283ce-ac3a-49f3-b327-18cec376ff01 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:38.821656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b944efd9-fef9-4115-af9d-0e53a32b5bae None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:38.839648 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e41c58-123a-4a4f-8be5-734ab1246158 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:38.840366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e41c58-123a-4a4f-8be5-734ab1246158 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:38.840646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e41c58-123a-4a4f-8be5-734ab1246158 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:38.840915 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e41c58-123a-4a4f-8be5-734ab1246158 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:38.849505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e41c58-123a-4a4f-8be5-734ab1246158 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:38.851127 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e41c58-123a-4a4f-8be5-734ab1246158 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:38.856742 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68db3895-8b9d-4b09-a649-c0ce9af98127 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:38.857354 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68db3895-8b9d-4b09-a649-c0ce9af98127 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:38.857634 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68db3895-8b9d-4b09-a649-c0ce9af98127 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:38.857904 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68db3895-8b9d-4b09-a649-c0ce9af98127 None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:38.858623 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68db3895-8b9d-4b09-a649-c0ce9af98127 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:38.858954 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68db3895-8b9d-4b09-a649-c0ce9af98127 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:38.860426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68db3895-8b9d-4b09-a649-c0ce9af98127 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:38.863004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-d4e41c58-123a-4a4f-8be5-734ab1246158 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 16 03:27:38.863004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:38.863004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:38.863004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:38.863004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:38.863004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:38.863004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:38.863004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:38.863004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:38.863004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:38.863004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:38.863004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:38.863004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:27:38.863004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:27:38.863004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:27:38.863004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:27:38.864571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:38.864571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:38.864571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:27:38.864571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:38.864571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:27:38.864571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:38.864571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:38.864571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:38.864571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:38.864571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:38.864571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:38.864571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:38.864571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:38.864571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:38.864571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:38.864571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:38.864571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:38.864571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:38.865802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:38.865802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:38.865802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:27:38.865802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:27:38.865802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:27:38.865802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:27:38.865802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:27:38.865802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:27:38.865802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 16 03:27:38.865802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:38.865802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 275/557] 10.222.0.49 () {56 vars in 1093 bytes} [Fri Sep 16 03:27:38 2022] GET /identity/v3/users/nova => generated 82 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:27:38.865802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbe30d0-7298-42cd-a895-d81fc84d8a0a None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:38.865802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbe30d0-7298-42cd-a895-d81fc84d8a0a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:38.866684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbe30d0-7298-42cd-a895-d81fc84d8a0a None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:38.867818 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 276/558] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:38 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:38.868065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbe30d0-7298-42cd-a895-d81fc84d8a0a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:38.883549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 283/559] 10.222.0.49 () {56 vars in 1135 bytes} [Fri Sep 16 03:27:38 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:38.884773 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea192c77-da85-4483-adf3-d946a99f481b None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:38.885056 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea192c77-da85-4483-adf3-d946a99f481b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:38.885327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea192c77-da85-4483-adf3-d946a99f481b None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:38.886666 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 284/560] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:38 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:38.887201 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea192c77-da85-4483-adf3-d946a99f481b None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:38.943616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:38.976993 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbe30d0-7298-42cd-a895-d81fc84d8a0a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:38.992946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea192c77-da85-4483-adf3-d946a99f481b None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:39.026169 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3193d919-28e9-4d57-8e47-7e7a715ce786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:39.026711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3193d919-28e9-4d57-8e47-7e7a715ce786 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:39.027027 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3193d919-28e9-4d57-8e47-7e7a715ce786 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:39.027304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3193d919-28e9-4d57-8e47-7e7a715ce786 None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:39.028019 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3193d919-28e9-4d57-8e47-7e7a715ce786 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:39.028322 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3193d919-28e9-4d57-8e47-7e7a715ce786 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:39.030019 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3193d919-28e9-4d57-8e47-7e7a715ce786 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:39.033823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:39.046655 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 285/561] 10.222.0.49 () {56 vars in 1133 bytes} [Fri Sep 16 03:27:38 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:39.048382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c24d2d-e1ef-40c9-a895-9a6dff1c2f1b None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:39.048660 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c24d2d-e1ef-40c9-a895-9a6dff1c2f1b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:39.048934 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c24d2d-e1ef-40c9-a895-9a6dff1c2f1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:39.053188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8dba548-1029-490e-9116-0330eb0aeaf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:39.053823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8dba548-1029-490e-9116-0330eb0aeaf4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:39.054105 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8dba548-1029-490e-9116-0330eb0aeaf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:39.054383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8dba548-1029-490e-9116-0330eb0aeaf4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:39.065428 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8dba548-1029-490e-9116-0330eb0aeaf4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:39.066411 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-10c24d2d-e1ef-40c9-a895-9a6dff1c2f1b None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:39.067633 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-10c24d2d-e1ef-40c9-a895-9a6dff1c2f1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:39.070239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-a8dba548-1029-490e-9116-0330eb0aeaf4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:39.070239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:39.070239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:39.070239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:39.070239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:39.070239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:39.070239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:39.070239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:39.070239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:39.070239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:39.070239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:39.070239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:39.070239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:39.070239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:39.070239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:39.070239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:39.076377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:39.076377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:39.076377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:39.076377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:39.076377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:39.076377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:39.076377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:39.076377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:39.076377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:39.076377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:39.076377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:39.076377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:39.076377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:39.076377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:39.076377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:39.076377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:39.078044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:39.078044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:39.078044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:39.078044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:39.078044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:39.078044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:39.078044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:39.078044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:39.078044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:39.078044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:39.078044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:39.078044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:39.078044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 277/562] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:27:38 2022] GET /identity/v3/roles/service => generated 85 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:39.078044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8dba548-1029-490e-9116-0330eb0aeaf4 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:39.134609 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 286/563] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:39.135908 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c732c7b5-c81c-4139-9da2-8ff06e49fac0 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:39.136194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c732c7b5-c81c-4139-9da2-8ff06e49fac0 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:39.136469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c732c7b5-c81c-4139-9da2-8ff06e49fac0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:39.154535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-c732c7b5-c81c-4139-9da2-8ff06e49fac0 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:39.156056 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c732c7b5-c81c-4139-9da2-8ff06e49fac0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:39.186350 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:39.191995 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8dba548-1029-490e-9116-0330eb0aeaf4 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:39.227383 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 287/564] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:39.228244 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c732c7b5-c81c-4139-9da2-8ff06e49fac0 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:39.238836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e16884-aa71-4c79-9fae-ab645fae6402 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:39.239752 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e16884-aa71-4c79-9fae-ab645fae6402 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:39.240231 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e16884-aa71-4c79-9fae-ab645fae6402 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:39.240695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e16884-aa71-4c79-9fae-ab645fae6402 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:39.290999 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:39.332783 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e16884-aa71-4c79-9fae-ab645fae6402 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:39.334477 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e16884-aa71-4c79-9fae-ab645fae6402 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:39.360341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c732c7b5-c81c-4139-9da2-8ff06e49fac0 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:39.405634 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3494afd5-2808-49e5-93ae-5f12f3d53bab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:39.406666 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3494afd5-2808-49e5-93ae-5f12f3d53bab None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:39.407056 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3494afd5-2808-49e5-93ae-5f12f3d53bab None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:39.407410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3494afd5-2808-49e5-93ae-5f12f3d53bab None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:39.420185 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3494afd5-2808-49e5-93ae-5f12f3d53bab None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:39.422862 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3494afd5-2808-49e5-93ae-5f12f3d53bab None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:39.439687 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 278/565] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:27:39.446493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e16884-aa71-4c79-9fae-ab645fae6402 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:39.446657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-3494afd5-2808-49e5-93ae-5f12f3d53bab None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:39.446657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:39.446657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:39.446657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:39.446657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:39.446657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:39.446657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:39.446657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:39.446657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:39.446657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:39.446657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:39.446657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:39.446657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:39.446657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:39.446657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:39.446657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:39.448116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:39.448116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:39.448116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:39.448116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:39.448116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:39.448116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:39.448116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:39.448116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:39.448116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:39.448116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:39.448116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:39.448116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:39.448116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:39.448116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:39.448116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:39.448116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:39.448116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:39.449007 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:39.449007 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:39.449007 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:39.449007 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 288/566] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:39.449007 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3494afd5-2808-49e5-93ae-5f12f3d53bab None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:39.472396 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:39.517282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:39.541985 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e16884-aa71-4c79-9fae-ab645fae6402 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:39.557070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3494afd5-2808-49e5-93ae-5f12f3d53bab None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:39.579428 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76bf915-5d1a-4bf7-bf09-565e3d40dca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:39.580034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76bf915-5d1a-4bf7-bf09-565e3d40dca4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:39.580317 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76bf915-5d1a-4bf7-bf09-565e3d40dca4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:39.580591 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76bf915-5d1a-4bf7-bf09-565e3d40dca4 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:39.581249 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76bf915-5d1a-4bf7-bf09-565e3d40dca4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:39.581552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76bf915-5d1a-4bf7-bf09-565e3d40dca4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:39.582927 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76bf915-5d1a-4bf7-bf09-565e3d40dca4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:39.590101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623349ff-9e43-48a6-99e2-8fa30a5887e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:39.590575 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 279/567] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:27:39 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:39.590721 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623349ff-9e43-48a6-99e2-8fa30a5887e8 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:39.591034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623349ff-9e43-48a6-99e2-8fa30a5887e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:39.591315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623349ff-9e43-48a6-99e2-8fa30a5887e8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:39.591421 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76bf915-5d1a-4bf7-bf09-565e3d40dca4 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:39.602708 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623349ff-9e43-48a6-99e2-8fa30a5887e8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:39.608133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-623349ff-9e43-48a6-99e2-8fa30a5887e8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:39.608133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:39.608133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:39.608133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:39.608133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:39.608133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:39.608133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:39.608133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:39.608133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:39.608133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:39.608133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:39.608133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:39.608133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:39.608133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:39.608133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:39.608133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:39.609187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:39.609187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:39.609187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:39.609187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:39.609187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:39.609187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:39.609187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:39.609187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:39.609187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:39.609187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:39.609187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:39.609187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:39.609187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:39.609187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:39.609187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:39.609187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:39.610608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:39.610608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:39.610608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:39.610608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:39.610608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:39.610608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:39.610608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:39.610608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:39.610608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:39.610608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:39.610608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:39.610608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:39.610608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 289/568] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:27:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:39.610608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623349ff-9e43-48a6-99e2-8fa30a5887e8 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:39.688777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:39.690665 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76bf915-5d1a-4bf7-bf09-565e3d40dca4 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:39.706573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623349ff-9e43-48a6-99e2-8fa30a5887e8 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:39.724146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8445b56a-fe70-43f4-832f-566fda17a251 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:39.724745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8445b56a-fe70-43f4-832f-566fda17a251 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:39.725021 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8445b56a-fe70-43f4-832f-566fda17a251 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:39.725292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8445b56a-fe70-43f4-832f-566fda17a251 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:39.725919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8445b56a-fe70-43f4-832f-566fda17a251 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:39.726216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8445b56a-fe70-43f4-832f-566fda17a251 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:39.727595 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8445b56a-fe70-43f4-832f-566fda17a251 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:39.729764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:39.739973 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 280/569] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:39.740577 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b701a5-54e4-4e65-a127-49c3cc71f99c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:39.741206 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b701a5-54e4-4e65-a127-49c3cc71f99c None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:39.741485 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b701a5-54e4-4e65-a127-49c3cc71f99c None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:39.741739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8445b56a-fe70-43f4-832f-566fda17a251 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:39.741843 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b701a5-54e4-4e65-a127-49c3cc71f99c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:39.752651 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b701a5-54e4-4e65-a127-49c3cc71f99c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:39.754096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b701a5-54e4-4e65-a127-49c3cc71f99c None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:39.768166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-38b701a5-54e4-4e65-a127-49c3cc71f99c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:39.768166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:39.768166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:39.768166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:39.768166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:39.768166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:39.768166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:39.768166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:39.768166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:39.768166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:39.768166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:39.768166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:39.768166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:39.768166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:39.768166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:39.768166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:39.769106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:39.769106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:39.769106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:39.769106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:39.769106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:39.769106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:39.769106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:39.769106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:39.769106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:39.769106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:39.769106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:39.769106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:39.769106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:39.769106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:39.769106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:39.769106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:39.769106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:39.770004 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:39.770004 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:39.770004 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:39.770004 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 290/570] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:39.770004 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b701a5-54e4-4e65-a127-49c3cc71f99c None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:39.844214 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8445b56a-fe70-43f4-832f-566fda17a251 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:39.868714 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b701a5-54e4-4e65-a127-49c3cc71f99c None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:39.878137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9be134-798e-4a93-a253-c549f3bd920c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:39.878769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9be134-798e-4a93-a253-c549f3bd920c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:39.879079 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9be134-798e-4a93-a253-c549f3bd920c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:39.879408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9be134-798e-4a93-a253-c549f3bd920c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:39.890105 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9be134-798e-4a93-a253-c549f3bd920c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:39.895892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-ba9be134-798e-4a93-a253-c549f3bd920c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:39.895892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:39.895892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:39.895892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:39.895892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:39.895892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:39.895892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:39.895892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:39.895892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:39.895892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:39.895892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:39.895892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:39.895892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:39.895892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:39.895892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:39.895892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:39.896839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:39.896839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:39.896839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:39.896839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:39.896839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:39.896839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:39.896839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:39.896839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:39.896839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:39.896839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:39.896839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:39.896839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:39.896839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:39.896839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:39.896839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:39.896839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:39.897777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:39.897777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:39.897777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:39.897777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:39.897777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:39.897777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:39.897777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:39.897777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:39.897777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:39.897777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:39.897777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:39.897777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:39.897777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 281/571] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:27:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:39.897777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9be134-798e-4a93-a253-c549f3bd920c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:39.906808 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:39.912447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac241e03-6841-4418-9053-1f9a6f1cc1c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:39.916184 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac241e03-6841-4418-9053-1f9a6f1cc1c5 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:39.916184 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac241e03-6841-4418-9053-1f9a6f1cc1c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:39.916184 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac241e03-6841-4418-9053-1f9a6f1cc1c5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:39.924602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac241e03-6841-4418-9053-1f9a6f1cc1c5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:39.926123 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac241e03-6841-4418-9053-1f9a6f1cc1c5 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:39.940535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-ac241e03-6841-4418-9053-1f9a6f1cc1c5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:39.940535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:39.940535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:39.940535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:39.940535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:39.940535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:39.940535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:39.940535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:39.940535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:39.940535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:39.940535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:39.940535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:39.940535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:39.940535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:39.940535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:39.940535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:39.945854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:39.945854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:39.945854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:39.945854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:39.945854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:39.945854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:39.945854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:39.945854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:39.945854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:39.945854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:39.945854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:39.945854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:39.945854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:39.945854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:39.945854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:39.945854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:39.945854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:39.946756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:39.946756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:39.946756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:39.946756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 291/572] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:39.946756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac241e03-6841-4418-9053-1f9a6f1cc1c5 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:39.961513 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:40.008295 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9be134-798e-4a93-a253-c549f3bd920c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:40.042237 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293b41dd-0b10-4966-848e-aa9e9343218d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:40.042845 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293b41dd-0b10-4966-848e-aa9e9343218d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:40.043173 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293b41dd-0b10-4966-848e-aa9e9343218d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:40.043452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293b41dd-0b10-4966-848e-aa9e9343218d None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:40.044139 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293b41dd-0b10-4966-848e-aa9e9343218d None admin] RBAC: Adding query filter params (name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:40.044442 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293b41dd-0b10-4966-848e-aa9e9343218d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:40.045808 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293b41dd-0b10-4966-848e-aa9e9343218d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:40.046630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac241e03-6841-4418-9053-1f9a6f1cc1c5 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:40.053528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 282/573] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:27:39 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:40.054312 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293b41dd-0b10-4966-848e-aa9e9343218d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:40.080757 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2a9a9a-37e5-4789-860a-0705d77e7437 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:40.081383 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2a9a9a-37e5-4789-860a-0705d77e7437 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:40.081661 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2a9a9a-37e5-4789-860a-0705d77e7437 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:40.081936 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2a9a9a-37e5-4789-860a-0705d77e7437 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:40.088500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef2a9a9a-37e5-4789-860a-0705d77e7437 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:40.089946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef2a9a9a-37e5-4789-860a-0705d77e7437 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:40.101044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-ef2a9a9a-37e5-4789-860a-0705d77e7437 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:40.101044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:40.101044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:40.101044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:40.101044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:40.101044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:40.101044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:40.101044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:40.101044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:40.101044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:40.101044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:40.101044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:40.101044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:27:40.101044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:27:40.101044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:27:40.101972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:27:40.101972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:40.101972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:40.101972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:40.101972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:40.101972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:40.101972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:40.101972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:40.101972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:40.101972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:40.101972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:40.101972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:40.101972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:40.101972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:40.101972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:40.101972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:40.101972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:40.102875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:27:40.102875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:27:40.102875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:27:40.102875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:27:40.102875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:27:40.102875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:27:40.102875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:40.102875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:40.102875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 292/574] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:40.102875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2a9a9a-37e5-4789-860a-0705d77e7437 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:40.147508 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:40.157129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293b41dd-0b10-4966-848e-aa9e9343218d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:40.174161 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:40.191010 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045bebea-3c10-424d-a305-32f9be1b51dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:40.191627 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045bebea-3c10-424d-a305-32f9be1b51dc None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:40.191904 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045bebea-3c10-424d-a305-32f9be1b51dc None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:40.192176 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045bebea-3c10-424d-a305-32f9be1b51dc None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:40.192813 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045bebea-3c10-424d-a305-32f9be1b51dc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:40.193112 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045bebea-3c10-424d-a305-32f9be1b51dc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:40.194482 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045bebea-3c10-424d-a305-32f9be1b51dc None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:40.199544 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2a9a9a-37e5-4789-860a-0705d77e7437 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:40.205866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 283/575] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:40.206645 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045bebea-3c10-424d-a305-32f9be1b51dc None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:40.235893 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4acb6c-e8a2-440a-baac-4de968d61793 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:40.236660 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4acb6c-e8a2-440a-baac-4de968d61793 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:40.236946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4acb6c-e8a2-440a-baac-4de968d61793 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:40.237222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4acb6c-e8a2-440a-baac-4de968d61793 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:40.241308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4acb6c-e8a2-440a-baac-4de968d61793 None admin] RBAC: Adding query filter params (name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:40.241624 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4acb6c-e8a2-440a-baac-4de968d61793 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:40.243954 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4acb6c-e8a2-440a-baac-4de968d61793 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:40.252366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 293/576] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:27:40 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:40.253163 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4acb6c-e8a2-440a-baac-4de968d61793 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:40.306760 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045bebea-3c10-424d-a305-32f9be1b51dc None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:40.345503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c234c22b-0c9a-4898-8b5a-c8a981f154e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:40.346112 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c234c22b-0c9a-4898-8b5a-c8a981f154e0 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:40.346393 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c234c22b-0c9a-4898-8b5a-c8a981f154e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:40.346669 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c234c22b-0c9a-4898-8b5a-c8a981f154e0 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:40.347344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c234c22b-0c9a-4898-8b5a-c8a981f154e0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:40.347650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c234c22b-0c9a-4898-8b5a-c8a981f154e0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:40.349018 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c234c22b-0c9a-4898-8b5a-c8a981f154e0 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:40.354140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4acb6c-e8a2-440a-baac-4de968d61793 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:40.360021 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 284/577] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:40.361145 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c234c22b-0c9a-4898-8b5a-c8a981f154e0 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:40.361349 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:40.391938 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a7961d-d8c1-4c0c-800d-d5d5e0ebf326 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:40.392534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a7961d-d8c1-4c0c-800d-d5d5e0ebf326 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:40.392820 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a7961d-d8c1-4c0c-800d-d5d5e0ebf326 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:40.393097 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a7961d-d8c1-4c0c-800d-d5d5e0ebf326 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:40.400073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:40.403604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a7961d-d8c1-4c0c-800d-d5d5e0ebf326 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:40.405430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a7961d-d8c1-4c0c-800d-d5d5e0ebf326 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:40.420574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-70a7961d-d8c1-4c0c-800d-d5d5e0ebf326 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:40.420574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:40.420574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:40.420574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:40.420574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:40.420574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:40.420574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:40.420574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:40.420574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:40.420574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:40.420574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:40.420574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:40.420574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:40.420574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:40.420574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:40.420574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:40.421614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:40.421614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:40.421614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:40.421614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:40.421614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:40.421614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:40.421614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:40.421614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:40.421614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:40.421614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:40.421614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:40.421614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:40.421614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:40.421614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:40.421614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:40.421614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:40.421614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:40.423248 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:40.423248 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:40.423248 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:40.427131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 294/578] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:40.427964 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a7961d-d8c1-4c0c-800d-d5d5e0ebf326 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:40.460465 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c234c22b-0c9a-4898-8b5a-c8a981f154e0 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:40.492949 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bd7958-e6a5-455f-8c87-164b9ebb6dff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:40.493545 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bd7958-e6a5-455f-8c87-164b9ebb6dff None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:40.493820 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bd7958-e6a5-455f-8c87-164b9ebb6dff None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:40.494091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bd7958-e6a5-455f-8c87-164b9ebb6dff None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:40.494773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bd7958-e6a5-455f-8c87-164b9ebb6dff None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:40.495141 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bd7958-e6a5-455f-8c87-164b9ebb6dff None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:40.496592 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bd7958-e6a5-455f-8c87-164b9ebb6dff None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:40.509038 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 285/579] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:27:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:40.509866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bd7958-e6a5-455f-8c87-164b9ebb6dff None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:40.527805 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a7961d-d8c1-4c0c-800d-d5d5e0ebf326 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:40.561501 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410fd8fc-6ce4-4b5b-b0c1-2aa438119de0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:40.562122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410fd8fc-6ce4-4b5b-b0c1-2aa438119de0 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:40.562403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410fd8fc-6ce4-4b5b-b0c1-2aa438119de0 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:40.562673 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410fd8fc-6ce4-4b5b-b0c1-2aa438119de0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:40.569396 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410fd8fc-6ce4-4b5b-b0c1-2aa438119de0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:40.570855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410fd8fc-6ce4-4b5b-b0c1-2aa438119de0 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:40.582527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-410fd8fc-6ce4-4b5b-b0c1-2aa438119de0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:40.582527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:40.582527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:40.582527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:40.582527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:40.582527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:40.582527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:40.582527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:40.582527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:40.582527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:40.582527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:40.582527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:40.582527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:27:40.582527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:27:40.582527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:27:40.583621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:27:40.583621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:40.583621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:40.583621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:40.583621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:40.583621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:40.583621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:40.583621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:40.583621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:40.583621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:40.583621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:40.583621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:40.583621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:40.583621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:40.583621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:40.583621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:40.583621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:40.584551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:27:40.584551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:27:40.584551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:27:40.584551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:27:40.584551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:27:40.584551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:27:40.584551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:40.584551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:40.584551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 295/580] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:40.585039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410fd8fc-6ce4-4b5b-b0c1-2aa438119de0 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:40.603450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:40.609635 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:40.613946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bd7958-e6a5-455f-8c87-164b9ebb6dff None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:40.647505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b187b6db-711d-462e-a8fd-464e9cf81360 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:40.648141 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b187b6db-711d-462e-a8fd-464e9cf81360 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:40.648422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b187b6db-711d-462e-a8fd-464e9cf81360 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:40.648697 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b187b6db-711d-462e-a8fd-464e9cf81360 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:40.658936 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b187b6db-711d-462e-a8fd-464e9cf81360 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:40.660378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b187b6db-711d-462e-a8fd-464e9cf81360 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:40.677062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-b187b6db-711d-462e-a8fd-464e9cf81360 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:40.677062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:40.677062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:40.677062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:40.677062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:40.677062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:40.677062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:40.677062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:40.677062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:40.677062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:40.677062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:40.677062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:40.677062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:40.677062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:40.677062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:40.677062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:40.678136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:40.678136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:40.678136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:40.678136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:40.678136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:40.678136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:40.678136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:40.678136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:40.678136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:40.678136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:40.678136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:40.678136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:40.678136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:40.678136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:40.678136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:40.678136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:40.678136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:40.679044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:40.679044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:40.679044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:40.682684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410fd8fc-6ce4-4b5b-b0c1-2aa438119de0 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:40.682944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 286/581] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:40.683773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b187b6db-711d-462e-a8fd-464e9cf81360 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:40.718178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf973d0c-edc6-454d-8cde-131c1bbb5b17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:40.718802 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf973d0c-edc6-454d-8cde-131c1bbb5b17 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:40.719257 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf973d0c-edc6-454d-8cde-131c1bbb5b17 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:40.720073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf973d0c-edc6-454d-8cde-131c1bbb5b17 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:40.728001 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf973d0c-edc6-454d-8cde-131c1bbb5b17 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:40.729459 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf973d0c-edc6-454d-8cde-131c1bbb5b17 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:40.740336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-cf973d0c-edc6-454d-8cde-131c1bbb5b17 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 16 03:27:40.740336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:40.740336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:40.740336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:40.740336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:40.740336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:40.740336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:40.740336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:40.740336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:40.740336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:40.740336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:40.740336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:40.740336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:27:40.740336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:27:40.740336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:27:40.740336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:27:40.741279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:40.741279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:40.741279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:27:40.741279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:40.741279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:27:40.741279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:40.741279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:40.741279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:40.741279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:40.741279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:40.741279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:40.741279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:40.741279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:40.741279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:40.741279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:40.741279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:40.741279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:40.741279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:40.742271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:40.742271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:40.742271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:27:40.742271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:27:40.742271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:27:40.742271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:27:40.742271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:27:40.742271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:27:40.742271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 16 03:27:40.742271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:40.742271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 296/582] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:40 2022] GET /identity/v3/users/glance => generated 84 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:40.742271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf973d0c-edc6-454d-8cde-131c1bbb5b17 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:40.786180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b187b6db-711d-462e-a8fd-464e9cf81360 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:40.817800 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:40.819077 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19db24d4-cc05-4daa-87f6-167a4461d739 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:40.819672 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19db24d4-cc05-4daa-87f6-167a4461d739 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:40.819948 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19db24d4-cc05-4daa-87f6-167a4461d739 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:40.820220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19db24d4-cc05-4daa-87f6-167a4461d739 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:40.820852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19db24d4-cc05-4daa-87f6-167a4461d739 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:40.821161 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19db24d4-cc05-4daa-87f6-167a4461d739 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:40.822509 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19db24d4-cc05-4daa-87f6-167a4461d739 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:40.834780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 287/583] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:40.835609 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19db24d4-cc05-4daa-87f6-167a4461d739 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:40.839487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf973d0c-edc6-454d-8cde-131c1bbb5b17 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:40.848357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:40.874159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df8e44b-d98c-490c-83b7-01b2d301ca6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:40.874759 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df8e44b-d98c-490c-83b7-01b2d301ca6f None admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:40.875061 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df8e44b-d98c-490c-83b7-01b2d301ca6f None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:40.875340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df8e44b-d98c-490c-83b7-01b2d301ca6f None admin] PATH_INFO: `/v3/projects/0bb7350474f349d6aee197ff35536167/users/ded1bcd10e62499a9aaa0ece5aa55c80/roles/974d839067e9411aa4cb850cee33f6ae` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:40.889371 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df8e44b-d98c-490c-83b7-01b2d301ca6f None admin] RBAC: Authorizing `identity:create_grant(project_id=0bb7350474f349d6aee197ff35536167, user_id=ded1bcd10e62499a9aaa0ece5aa55c80, role_id=974d839067e9411aa4cb850cee33f6ae)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:40.890814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df8e44b-d98c-490c-83b7-01b2d301ca6f None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:40.938887 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19db24d4-cc05-4daa-87f6-167a4461d739 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:40.971670 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 297/584] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:27:40 2022] PUT /identity/v3/projects/0bb7350474f349d6aee197ff35536167/users/ded1bcd10e62499a9aaa0ece5aa55c80/roles/974d839067e9411aa4cb850cee33f6ae => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:27:40.972569 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df8e44b-d98c-490c-83b7-01b2d301ca6f None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:40.972809 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a6f267-660d-4c60-8fec-b2ca44a27399 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:40.973194 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a6f267-660d-4c60-8fec-b2ca44a27399 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:40.973474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a6f267-660d-4c60-8fec-b2ca44a27399 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:40.973745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a6f267-660d-4c60-8fec-b2ca44a27399 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:40.974440 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a6f267-660d-4c60-8fec-b2ca44a27399 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:40.974748 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a6f267-660d-4c60-8fec-b2ca44a27399 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:40.976235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a6f267-660d-4c60-8fec-b2ca44a27399 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:40.989316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 288/585] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:27:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:40.990189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a6f267-660d-4c60-8fec-b2ca44a27399 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:41.069786 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:41.098951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df8e44b-d98c-490c-83b7-01b2d301ca6f None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:41.102800 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a6f267-660d-4c60-8fec-b2ca44a27399 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:41.131061 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:41.139469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb4e3d0-9a19-4c20-934b-f169ee567be6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:41.140146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a47681e-cf9c-4a87-9fc3-406bd1f78a81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:41.142560 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb4e3d0-9a19-4c20-934b-f169ee567be6 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:41.142560 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb4e3d0-9a19-4c20-934b-f169ee567be6 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:41.142755 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a47681e-cf9c-4a87-9fc3-406bd1f78a81 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:41.142837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb4e3d0-9a19-4c20-934b-f169ee567be6 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:41.142955 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a47681e-cf9c-4a87-9fc3-406bd1f78a81 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:41.142955 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a47681e-cf9c-4a87-9fc3-406bd1f78a81 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:41.143109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb4e3d0-9a19-4c20-934b-f169ee567be6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:41.143109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb4e3d0-9a19-4c20-934b-f169ee567be6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:41.143254 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a47681e-cf9c-4a87-9fc3-406bd1f78a81 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:41.143254 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a47681e-cf9c-4a87-9fc3-406bd1f78a81 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:41.144039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb4e3d0-9a19-4c20-934b-f169ee567be6 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:41.144153 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a47681e-cf9c-4a87-9fc3-406bd1f78a81 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:41.157928 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 298/586] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:41.159006 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb4e3d0-9a19-4c20-934b-f169ee567be6 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:41.174492 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 289/587] 10.222.0.49 () {56 vars in 1137 bytes} [Fri Sep 16 03:27:40 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:41.175669 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a47681e-cf9c-4a87-9fc3-406bd1f78a81 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:41.268557 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb4e3d0-9a19-4c20-934b-f169ee567be6 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:41.289491 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a47681e-cf9c-4a87-9fc3-406bd1f78a81 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:41.304369 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef641a8-05c2-4bbe-b7fb-1c8585c46596 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:41.305186 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef641a8-05c2-4bbe-b7fb-1c8585c46596 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:41.305568 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef641a8-05c2-4bbe-b7fb-1c8585c46596 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:41.305853 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef641a8-05c2-4bbe-b7fb-1c8585c46596 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:41.315923 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef641a8-05c2-4bbe-b7fb-1c8585c46596 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:41.317385 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef641a8-05c2-4bbe-b7fb-1c8585c46596 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:41.321700 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:41.333282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708b02a0-a830-4c10-9176-5c2b6b155343 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:41.339610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-eef641a8-05c2-4bbe-b7fb-1c8585c46596 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 16 03:27:41.339610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:41.339610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:41.339610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:41.339610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:41.339610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:41.339610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:41.339610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:41.339610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:41.339610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:41.339610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:41.339610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:41.339610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:27:41.339610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:27:41.339610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:27:41.339610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:27:41.340709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:41.340709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:41.340709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:27:41.340709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:41.340709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:27:41.340709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:41.340709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:41.340709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:41.340709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:41.340709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:41.340709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:41.340709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:41.340709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:41.340709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:41.340709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:41.340709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:41.340709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:41.340709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:41.341912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:41.341912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:41.341912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:27:41.341912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:27:41.341912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:27:41.341912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:27:41.341912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:27:41.341912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:27:41.341912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 16 03:27:41.341912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:41.342563 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708b02a0-a830-4c10-9176-5c2b6b155343 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:41.342563 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708b02a0-a830-4c10-9176-5c2b6b155343 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:41.342563 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708b02a0-a830-4c10-9176-5c2b6b155343 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:41.342809 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 299/588] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:27:41 2022] GET /identity/v3/users/neutron => generated 85 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:41.342894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708b02a0-a830-4c10-9176-5c2b6b155343 None admin] RBAC: Adding query filter params (role.id=974d839067e9411aa4cb850cee33f6ae, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=aa30da9972e140d7905c522576681429) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:41.342894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708b02a0-a830-4c10-9176-5c2b6b155343 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:41.343090 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef641a8-05c2-4bbe-b7fb-1c8585c46596 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:41.343174 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708b02a0-a830-4c10-9176-5c2b6b155343 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:41.355245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 290/589] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:27:41 2022] GET /identity/v3/role_assignments?user.id=aa30da9972e140d7905c522576681429&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=974d839067e9411aa4cb850cee33f6ae => generated 259 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:41.355245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708b02a0-a830-4c10-9176-5c2b6b155343 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:41.399886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:41.478678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef641a8-05c2-4bbe-b7fb-1c8585c46596 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:41.493981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708b02a0-a830-4c10-9176-5c2b6b155343 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:41.519429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804416e9-7cd2-48bd-bbeb-76daf12c56ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:41.520322 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804416e9-7cd2-48bd-bbeb-76daf12c56ba None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:41.521426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804416e9-7cd2-48bd-bbeb-76daf12c56ba None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:41.521855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804416e9-7cd2-48bd-bbeb-76daf12c56ba None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:41.532122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804416e9-7cd2-48bd-bbeb-76daf12c56ba None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:41.534397 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804416e9-7cd2-48bd-bbeb-76daf12c56ba None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:41.541544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1413063-475c-472a-9730-ed20b7038442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:41.542188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1413063-475c-472a-9730-ed20b7038442 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:41.542468 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1413063-475c-472a-9730-ed20b7038442 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:41.542739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1413063-475c-472a-9730-ed20b7038442 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:41.554722 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1413063-475c-472a-9730-ed20b7038442 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:41.556219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1413063-475c-472a-9730-ed20b7038442 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:41.558085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-804416e9-7cd2-48bd-bbeb-76daf12c56ba None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 16 03:27:41.558085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:41.558085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:41.558085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:41.558085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:41.558085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:41.558085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:41.558085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:41.558085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:41.558085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:41.558085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:41.558085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:41.558085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:27:41.558085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:27:41.558085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:27:41.558085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:27:41.559245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:41.559245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:41.559245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:27:41.559245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:41.559245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:27:41.559245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:41.559245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:41.559245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:41.559245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:41.559245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:41.559245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:41.559245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:41.559245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:41.559245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:41.559245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:41.559245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:41.559245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:41.559245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:41.560367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:41.560367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:41.560367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:27:41.560367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:27:41.560367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:27:41.560367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:27:41.560367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:27:41.560367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:27:41.560367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 16 03:27:41.560367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:41.562568 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 300/590] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:41 2022] GET /identity/v3/users/cinder => generated 84 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:41.563888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804416e9-7cd2-48bd-bbeb-76daf12c56ba None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:41.571068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-b1413063-475c-472a-9730-ed20b7038442 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:41.571068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:41.571068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:41.571068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:41.571068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:41.571068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:41.571068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:41.571068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:41.571068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:41.571068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:41.571068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:41.571068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:41.571068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:41.571068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:41.571068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:41.571068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:41.572241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:41.572241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:41.572241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:41.572241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:41.572241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:41.572241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:41.572241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:41.572241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:41.572241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:41.572241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:41.572241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:41.572241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:41.572241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:41.572241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:41.572241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:41.572241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:41.572241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:41.573527 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:41.573527 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:41.573527 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:41.576881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 291/591] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:41.577966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1413063-475c-472a-9730-ed20b7038442 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:41.627582 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:41.689359 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804416e9-7cd2-48bd-bbeb-76daf12c56ba None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:41.691972 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1413063-475c-472a-9730-ed20b7038442 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:41.700910 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:41.731653 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca03f90e-ce5b-40c6-951e-ca682bb94106 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:41.732271 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca03f90e-ce5b-40c6-951e-ca682bb94106 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:41.732551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca03f90e-ce5b-40c6-951e-ca682bb94106 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:41.732826 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca03f90e-ce5b-40c6-951e-ca682bb94106 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:41.733562 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca03f90e-ce5b-40c6-951e-ca682bb94106 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:41.733873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca03f90e-ce5b-40c6-951e-ca682bb94106 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:41.733987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056abac5-3e3d-408b-b3c4-07b0e9fb7488 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:41.734568 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056abac5-3e3d-408b-b3c4-07b0e9fb7488 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:41.734921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056abac5-3e3d-408b-b3c4-07b0e9fb7488 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:41.735205 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056abac5-3e3d-408b-b3c4-07b0e9fb7488 None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:41.735473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca03f90e-ce5b-40c6-951e-ca682bb94106 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:41.735927 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056abac5-3e3d-408b-b3c4-07b0e9fb7488 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:41.736229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056abac5-3e3d-408b-b3c4-07b0e9fb7488 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:41.737937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056abac5-3e3d-408b-b3c4-07b0e9fb7488 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:41.755692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 292/592] 10.222.0.49 () {56 vars in 1137 bytes} [Fri Sep 16 03:27:41 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:41.757104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca03f90e-ce5b-40c6-951e-ca682bb94106 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:41.759691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 301/593] 10.222.0.49 () {56 vars in 1139 bytes} [Fri Sep 16 03:27:41 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:41.764109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056abac5-3e3d-408b-b3c4-07b0e9fb7488 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:41.878986 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca03f90e-ce5b-40c6-951e-ca682bb94106 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:41.880972 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:41.898940 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056abac5-3e3d-408b-b3c4-07b0e9fb7488 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:41.925290 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c267b3-da93-466d-ae15-52a559e3f4bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:41.926153 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c267b3-da93-466d-ae15-52a559e3f4bd None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:41.926436 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c267b3-da93-466d-ae15-52a559e3f4bd None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:41.926708 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c267b3-da93-466d-ae15-52a559e3f4bd None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:41.927987 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c267b3-da93-466d-ae15-52a559e3f4bd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:41.928416 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c267b3-da93-466d-ae15-52a559e3f4bd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:41.929889 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c267b3-da93-466d-ae15-52a559e3f4bd None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:41.937825 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d84185-e7a2-4912-9dfd-ad23ba3e7396 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:41.939262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d84185-e7a2-4912-9dfd-ad23ba3e7396 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:41.939638 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d84185-e7a2-4912-9dfd-ad23ba3e7396 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:41.939824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d84185-e7a2-4912-9dfd-ad23ba3e7396 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:41.944210 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 293/594] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:41.949717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c267b3-da93-466d-ae15-52a559e3f4bd None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:41.955241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d84185-e7a2-4912-9dfd-ad23ba3e7396 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:41.956809 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d84185-e7a2-4912-9dfd-ad23ba3e7396 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:41.972452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-d1d84185-e7a2-4912-9dfd-ad23ba3e7396 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:41.972452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:41.972452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:41.972452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:41.972452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:41.972452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:41.972452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:41.972452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:41.972452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:41.972452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:41.972452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:41.972452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:41.972452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:41.972452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:41.972452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:41.972452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:41.975054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:41.975054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:41.975054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:41.975054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:41.975054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:41.975054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:41.975054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:41.975054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:41.975054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:41.975054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:41.975054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:41.975054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:41.975054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:41.975054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:41.975054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:41.975054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:41.975054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:41.977160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:41.977160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:41.977160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:41.977160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 302/595] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:27:41.977160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d84185-e7a2-4912-9dfd-ad23ba3e7396 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:41.994444 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:42.059366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c267b3-da93-466d-ae15-52a559e3f4bd None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:42.094958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6241a0f-8f2f-48a1-9b6e-062f26c47ebb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:42.095612 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6241a0f-8f2f-48a1-9b6e-062f26c47ebb None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:42.095897 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6241a0f-8f2f-48a1-9b6e-062f26c47ebb None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:42.096178 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6241a0f-8f2f-48a1-9b6e-062f26c47ebb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:42.096342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d84185-e7a2-4912-9dfd-ad23ba3e7396 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:42.108282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6241a0f-8f2f-48a1-9b6e-062f26c47ebb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:42.110404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6241a0f-8f2f-48a1-9b6e-062f26c47ebb None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:42.126070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-d6241a0f-8f2f-48a1-9b6e-062f26c47ebb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:42.126070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:42.126070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:42.126070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:42.126070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:42.126070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:42.126070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:42.126070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:42.126070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:42.126070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:42.126070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:42.126070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:42.126070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:42.126070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:42.126070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:42.126070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:42.128446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:42.128446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:42.128446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:42.128446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:42.128446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:42.128446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:42.128446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:42.128446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:42.128446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:42.128446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:42.128446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:42.128446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:42.128446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:42.128446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:42.128446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:42.128446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:42.128446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:42.130621 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:42.130621 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:42.130621 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:42.130621 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 294/596] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:42.130621 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6241a0f-8f2f-48a1-9b6e-062f26c47ebb None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:42.139565 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10711a00-7499-4ec4-a960-cc8df41e3c8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:42.140206 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10711a00-7499-4ec4-a960-cc8df41e3c8c None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:42.140486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10711a00-7499-4ec4-a960-cc8df41e3c8c None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:42.140758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10711a00-7499-4ec4-a960-cc8df41e3c8c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:42.150098 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:42.152212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10711a00-7499-4ec4-a960-cc8df41e3c8c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:42.154278 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10711a00-7499-4ec4-a960-cc8df41e3c8c None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:42.169468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-10711a00-7499-4ec4-a960-cc8df41e3c8c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:42.169468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:42.169468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:42.169468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:42.169468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:42.169468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:42.169468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:42.169468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:42.169468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:42.169468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:42.169468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:42.169468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:42.169468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:27:42.169468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:27:42.169468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:27:42.170571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:27:42.170571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:42.170571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:42.170571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:42.170571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:42.170571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:42.170571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:42.170571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:42.170571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:42.170571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:42.170571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:42.170571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:42.170571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:42.170571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:42.170571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:42.170571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:42.170571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:42.171787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:27:42.171787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:27:42.171787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:27:42.171787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:27:42.171787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:27:42.171787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:27:42.171787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:42.171787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:42.174782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 303/597] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:42.175748 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10711a00-7499-4ec4-a960-cc8df41e3c8c None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:42.248378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6241a0f-8f2f-48a1-9b6e-062f26c47ebb None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:42.281522 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:42.292453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31018933-3a19-43e0-8f7d-330a5af6e635 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:42.305902 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31018933-3a19-43e0-8f7d-330a5af6e635 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:42.305902 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31018933-3a19-43e0-8f7d-330a5af6e635 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:42.305902 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31018933-3a19-43e0-8f7d-330a5af6e635 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:42.305902 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31018933-3a19-43e0-8f7d-330a5af6e635 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:42.305902 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31018933-3a19-43e0-8f7d-330a5af6e635 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:42.305902 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31018933-3a19-43e0-8f7d-330a5af6e635 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:42.314699 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 295/598] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:42.315790 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31018933-3a19-43e0-8f7d-330a5af6e635 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:42.319132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10711a00-7499-4ec4-a960-cc8df41e3c8c None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:42.369864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6564bd3-cf72-49f7-bf8e-d726ecff8d82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:42.370470 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6564bd3-cf72-49f7-bf8e-d726ecff8d82 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:42.370747 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6564bd3-cf72-49f7-bf8e-d726ecff8d82 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:42.371084 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6564bd3-cf72-49f7-bf8e-d726ecff8d82 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:42.371947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6564bd3-cf72-49f7-bf8e-d726ecff8d82 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:42.372261 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6564bd3-cf72-49f7-bf8e-d726ecff8d82 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:42.374021 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6564bd3-cf72-49f7-bf8e-d726ecff8d82 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:42.394497 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 304/599] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:42.394497 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6564bd3-cf72-49f7-bf8e-d726ecff8d82 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:42.406745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:42.435232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31018933-3a19-43e0-8f7d-330a5af6e635 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:42.478539 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82eb365c-8e99-47ee-818d-99472078b8c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:42.479177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82eb365c-8e99-47ee-818d-99472078b8c1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:42.479463 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82eb365c-8e99-47ee-818d-99472078b8c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:42.479737 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82eb365c-8e99-47ee-818d-99472078b8c1 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:42.480874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82eb365c-8e99-47ee-818d-99472078b8c1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:42.481188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82eb365c-8e99-47ee-818d-99472078b8c1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:42.483024 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82eb365c-8e99-47ee-818d-99472078b8c1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:42.505078 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 296/600] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:27:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:42.505078 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82eb365c-8e99-47ee-818d-99472078b8c1 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:42.511034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6564bd3-cf72-49f7-bf8e-d726ecff8d82 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:42.548136 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6ed7f4-cd08-419b-a750-483e29c4e4c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:42.549256 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6ed7f4-cd08-419b-a750-483e29c4e4c1 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:42.549743 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6ed7f4-cd08-419b-a750-483e29c4e4c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:42.550238 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6ed7f4-cd08-419b-a750-483e29c4e4c1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:42.558161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6ed7f4-cd08-419b-a750-483e29c4e4c1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:42.559943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6ed7f4-cd08-419b-a750-483e29c4e4c1 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:42.562114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:42.573616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-5a6ed7f4-cd08-419b-a750-483e29c4e4c1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:42.573616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:42.573616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:42.573616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:42.573616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:42.573616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:42.573616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:42.573616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:42.573616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:42.573616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:42.573616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:42.573616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:42.573616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:27:42.573616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:27:42.573616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:27:42.577855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:27:42.577855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:42.577855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:42.577855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:42.577855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:42.577855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:42.577855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:42.577855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:42.577855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:42.577855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:42.577855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:42.577855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:42.577855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:42.577855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:42.577855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:42.577855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:42.577855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:42.579358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:27:42.579358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:27:42.579358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:27:42.579358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:27:42.579358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:27:42.579358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:27:42.579358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:42.579358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:42.579358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 305/601] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:42.579358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6ed7f4-cd08-419b-a750-483e29c4e4c1 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:42.626000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82eb365c-8e99-47ee-818d-99472078b8c1 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:42.668757 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9429e71e-f2aa-4cd7-abcf-cf226bd57f91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:42.669663 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9429e71e-f2aa-4cd7-abcf-cf226bd57f91 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:42.669945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9429e71e-f2aa-4cd7-abcf-cf226bd57f91 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:42.670221 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9429e71e-f2aa-4cd7-abcf-cf226bd57f91 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:42.673162 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:42.679508 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9429e71e-f2aa-4cd7-abcf-cf226bd57f91 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:42.683370 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9429e71e-f2aa-4cd7-abcf-cf226bd57f91 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:42.695890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-9429e71e-f2aa-4cd7-abcf-cf226bd57f91 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:42.695890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:42.695890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:42.695890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:42.695890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:42.695890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:42.695890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:42.695890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:42.695890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:42.695890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:42.695890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:42.695890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:42.695890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:27:42.695890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:27:42.695890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:27:42.696804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:27:42.696804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:42.696804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:42.696804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:42.696804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:42.696804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:42.696804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:42.696804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:42.696804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:42.696804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:42.696804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:42.696804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:42.696804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:42.696804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:42.696804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:42.696804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:42.696804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:42.697700 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:27:42.697700 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:27:42.697700 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:27:42.697700 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:27:42.697700 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:27:42.697700 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:27:42.697700 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:42.697700 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:42.704710 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 297/602] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:42.705849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9429e71e-f2aa-4cd7-abcf-cf226bd57f91 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:42.712353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6ed7f4-cd08-419b-a750-483e29c4e4c1 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:42.747306 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ac9d84-c6d9-4291-8324-c1a29129016d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:42.747961 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ac9d84-c6d9-4291-8324-c1a29129016d None admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:42.748240 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ac9d84-c6d9-4291-8324-c1a29129016d None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:42.748513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ac9d84-c6d9-4291-8324-c1a29129016d None admin] PATH_INFO: `/v3/projects/0bb7350474f349d6aee197ff35536167/users/05ceb9fc31c345ccabaee8f30a3982a8/roles/974d839067e9411aa4cb850cee33f6ae` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:42.764324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ac9d84-c6d9-4291-8324-c1a29129016d None admin] RBAC: Authorizing `identity:create_grant(project_id=0bb7350474f349d6aee197ff35536167, user_id=05ceb9fc31c345ccabaee8f30a3982a8, role_id=974d839067e9411aa4cb850cee33f6ae)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:42.765460 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ac9d84-c6d9-4291-8324-c1a29129016d None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:42.813982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9429e71e-f2aa-4cd7-abcf-cf226bd57f91 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:42.848543 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 306/603] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:27:42 2022] PUT /identity/v3/projects/0bb7350474f349d6aee197ff35536167/users/05ceb9fc31c345ccabaee8f30a3982a8/roles/974d839067e9411aa4cb850cee33f6ae => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:27:42.848543 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ac9d84-c6d9-4291-8324-c1a29129016d None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:42.862100 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3ace18-0f2a-4fea-ba62-dd080a6eba2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:42.862875 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3ace18-0f2a-4fea-ba62-dd080a6eba2b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:42.863190 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3ace18-0f2a-4fea-ba62-dd080a6eba2b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:42.863467 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3ace18-0f2a-4fea-ba62-dd080a6eba2b None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:42.864186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3ace18-0f2a-4fea-ba62-dd080a6eba2b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:42.864496 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3ace18-0f2a-4fea-ba62-dd080a6eba2b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:42.865975 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3ace18-0f2a-4fea-ba62-dd080a6eba2b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:42.893793 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 298/604] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:27:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:42.899438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3ace18-0f2a-4fea-ba62-dd080a6eba2b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:42.961651 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:43.011758 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:43.072632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ac9d84-c6d9-4291-8324-c1a29129016d None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:43.096238 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3ace18-0f2a-4fea-ba62-dd080a6eba2b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:43.120743 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4af6c7-8a17-44a7-b826-35c4479d7c69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:43.121761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4af6c7-8a17-44a7-b826-35c4479d7c69 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:43.121761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4af6c7-8a17-44a7-b826-35c4479d7c69 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:43.121959 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4af6c7-8a17-44a7-b826-35c4479d7c69 None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:43.122588 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4af6c7-8a17-44a7-b826-35c4479d7c69 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:43.122980 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4af6c7-8a17-44a7-b826-35c4479d7c69 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:43.124553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4af6c7-8a17-44a7-b826-35c4479d7c69 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:43.138470 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 307/605] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:27:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:43.139274 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52dfd73-badc-4bdf-af12-5caddeed7176 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:43.140704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52dfd73-badc-4bdf-af12-5caddeed7176 None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:43.141928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52dfd73-badc-4bdf-af12-5caddeed7176 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:43.142495 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52dfd73-badc-4bdf-af12-5caddeed7176 None admin] PATH_INFO: `/v3/projects/0bb7350474f349d6aee197ff35536167/users/e6aeffd924734771bc5bf0d282c9052d/roles/974d839067e9411aa4cb850cee33f6ae` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:43.145074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4af6c7-8a17-44a7-b826-35c4479d7c69 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:43.161018 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52dfd73-badc-4bdf-af12-5caddeed7176 None admin] RBAC: Authorizing `identity:create_grant(project_id=0bb7350474f349d6aee197ff35536167, user_id=e6aeffd924734771bc5bf0d282c9052d, role_id=974d839067e9411aa4cb850cee33f6ae)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:43.163432 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52dfd73-badc-4bdf-af12-5caddeed7176 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:43.259536 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4af6c7-8a17-44a7-b826-35c4479d7c69 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:43.271503 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:43.275111 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 299/606] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:27:42 2022] PUT /identity/v3/projects/0bb7350474f349d6aee197ff35536167/users/e6aeffd924734771bc5bf0d282c9052d/roles/974d839067e9411aa4cb850cee33f6ae => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:27:43.283699 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378b04e9-2d8f-46e6-b01b-9e6fc0c3d48f None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:43.284529 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378b04e9-2d8f-46e6-b01b-9e6fc0c3d48f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:43.285087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378b04e9-2d8f-46e6-b01b-9e6fc0c3d48f None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:43.287406 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 300/607] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:43 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:43.293434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cc9435-811b-4bed-8c46-9ba7abfe7293 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:43.294522 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cc9435-811b-4bed-8c46-9ba7abfe7293 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:43.295028 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cc9435-811b-4bed-8c46-9ba7abfe7293 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:43.330272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1844e1a3-220b-4760-a3a6-b9a5bf527c76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:43.331216 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1844e1a3-220b-4760-a3a6-b9a5bf527c76 None admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:43.331502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1844e1a3-220b-4760-a3a6-b9a5bf527c76 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:43.331777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1844e1a3-220b-4760-a3a6-b9a5bf527c76 None admin] PATH_INFO: `/v3/projects/0bb7350474f349d6aee197ff35536167/users/f9275eeba471423e9fa2a31a6e08676d/roles/974d839067e9411aa4cb850cee33f6ae` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:43.342654 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-04cc9435-811b-4bed-8c46-9ba7abfe7293 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:43.343964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-04cc9435-811b-4bed-8c46-9ba7abfe7293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:43.367585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1844e1a3-220b-4760-a3a6-b9a5bf527c76 None admin] RBAC: Authorizing `identity:create_grant(project_id=0bb7350474f349d6aee197ff35536167, user_id=f9275eeba471423e9fa2a31a6e08676d, role_id=974d839067e9411aa4cb850cee33f6ae)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:43.369073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1844e1a3-220b-4760-a3a6-b9a5bf527c76 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:43.446383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 301/608] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:43.462184 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 308/609] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:27:43 2022] PUT /identity/v3/projects/0bb7350474f349d6aee197ff35536167/users/f9275eeba471423e9fa2a31a6e08676d/roles/974d839067e9411aa4cb850cee33f6ae => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:27:43.654445 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd44da0-4eb2-4a20-aa7c-da77407826da None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:43.655378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd44da0-4eb2-4a20-aa7c-da77407826da None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:43.655863 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd44da0-4eb2-4a20-aa7c-da77407826da None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:43.658631 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 302/610] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:43 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:43.664260 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1781d15e-f812-4e86-9e27-99138a4d9edd None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:43.664549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1781d15e-f812-4e86-9e27-99138a4d9edd None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:43.664827 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1781d15e-f812-4e86-9e27-99138a4d9edd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:43.689004 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-1781d15e-f812-4e86-9e27-99138a4d9edd None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:43.690379 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1781d15e-f812-4e86-9e27-99138a4d9edd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:43.759938 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd44da0-4eb2-4a20-aa7c-da77407826da None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:43.770032 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 309/611] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:43.799834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:43.867784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd44da0-4eb2-4a20-aa7c-da77407826da None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:43.903102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8146e3c6-ea6d-40d0-b5db-2110117ce23f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:43.903730 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8146e3c6-ea6d-40d0-b5db-2110117ce23f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:43.904003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8146e3c6-ea6d-40d0-b5db-2110117ce23f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:43.904280 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8146e3c6-ea6d-40d0-b5db-2110117ce23f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:43.925813 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8146e3c6-ea6d-40d0-b5db-2110117ce23f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:43.934969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-8146e3c6-ea6d-40d0-b5db-2110117ce23f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:43.934969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:43.934969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:43.934969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:43.934969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:43.934969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:43.934969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:43.934969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:43.934969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:43.934969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:43.934969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:43.934969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:43.934969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:43.934969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:43.934969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:43.934969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:43.935970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:43.935970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:43.935970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:43.935970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:43.935970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:43.935970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:43.935970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:43.935970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:43.935970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:43.935970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:43.935970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:43.935970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:43.935970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:43.935970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:43.935970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:43.935970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:43.936836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:43.936836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:43.936836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:43.936836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:43.936836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:43.936836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:43.936836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:43.936836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:43.936836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:43.936836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:43.936836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:43.936836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:43.936836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 303/612] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:27:43 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:27:43.944046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1781d15e-f812-4e86-9e27-99138a4d9edd None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:44.073041 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:44.088721 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1781d15e-f812-4e86-9e27-99138a4d9edd None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:44.136521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4caab94-5c0f-4483-b734-5649fd69be6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:44.137134 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4caab94-5c0f-4483-b734-5649fd69be6d None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:44.137418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4caab94-5c0f-4483-b734-5649fd69be6d None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:44.137706 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4caab94-5c0f-4483-b734-5649fd69be6d None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:44.138411 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4caab94-5c0f-4483-b734-5649fd69be6d None admin] RBAC: Adding query filter params (name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:44.138721 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4caab94-5c0f-4483-b734-5649fd69be6d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:44.140287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4caab94-5c0f-4483-b734-5649fd69be6d None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:44.153444 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 310/613] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:27:43 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:44.160320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8146e3c6-ea6d-40d0-b5db-2110117ce23f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:44.202644 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4caab94-5c0f-4483-b734-5649fd69be6d None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:44.280362 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:44.288962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8146e3c6-ea6d-40d0-b5db-2110117ce23f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:44.332560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47163eda-2fc4-400d-919f-0bbe7e83c5ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:44.333803 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47163eda-2fc4-400d-919f-0bbe7e83c5ee None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:44.334218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47163eda-2fc4-400d-919f-0bbe7e83c5ee None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:44.334639 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47163eda-2fc4-400d-919f-0bbe7e83c5ee None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:44.354226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47163eda-2fc4-400d-919f-0bbe7e83c5ee None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:44.355737 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47163eda-2fc4-400d-919f-0bbe7e83c5ee None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:44.384551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-47163eda-2fc4-400d-919f-0bbe7e83c5ee None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:44.384551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:44.384551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:44.384551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:44.384551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:44.384551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:44.384551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:44.384551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:44.384551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:44.384551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:44.384551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:44.384551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:44.384551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:44.384551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:44.384551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:44.384551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:44.385573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:44.385573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:44.385573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:44.385573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:44.385573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:44.385573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:44.385573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:44.385573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:44.385573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:44.385573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:44.385573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:44.385573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:44.385573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:44.385573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:44.385573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:44.385573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:44.385573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:44.386449 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:44.386449 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:44.386449 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:44.391575 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 304/614] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:44.397448 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47163eda-2fc4-400d-919f-0bbe7e83c5ee None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:44.402172 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4caab94-5c0f-4483-b734-5649fd69be6d None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:44.468401 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b6089f-4358-4856-9a56-e78fec451450 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:44.468954 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b6089f-4358-4856-9a56-e78fec451450 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:44.469290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b6089f-4358-4856-9a56-e78fec451450 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:44.469551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b6089f-4358-4856-9a56-e78fec451450 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:44.492599 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b6089f-4358-4856-9a56-e78fec451450 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:44.505463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-e5b6089f-4358-4856-9a56-e78fec451450 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:44.505463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:44.505463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:44.505463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:44.505463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:44.505463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:44.505463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:44.505463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:44.505463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:44.505463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:44.505463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:44.505463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:44.505463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:44.505463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:44.505463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:44.505463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:44.506639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:44.506639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:44.506639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:44.506639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:44.506639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:44.506639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:44.506639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:44.506639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:44.506639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:44.506639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:44.506639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:44.506639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:44.506639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:44.506639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:44.506639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:44.506639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:44.507668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:44.507668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:44.507668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:44.507668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:44.507668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:44.507668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:44.507668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:44.507668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:44.507668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:44.507668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:44.507668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:44.507668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:44.507668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 311/615] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:27:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:44.513451 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b6089f-4358-4856-9a56-e78fec451450 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:44.534433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:44.556081 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47163eda-2fc4-400d-919f-0bbe7e83c5ee None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:44.617631 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e731cb9-2c18-4e8e-86e0-78b5b1258e31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:44.618236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e731cb9-2c18-4e8e-86e0-78b5b1258e31 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:44.618515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e731cb9-2c18-4e8e-86e0-78b5b1258e31 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:44.618783 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e731cb9-2c18-4e8e-86e0-78b5b1258e31 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:44.619448 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e731cb9-2c18-4e8e-86e0-78b5b1258e31 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:44.619748 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e731cb9-2c18-4e8e-86e0-78b5b1258e31 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:44.621091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e731cb9-2c18-4e8e-86e0-78b5b1258e31 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:44.646361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 305/616] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:44.652067 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e731cb9-2c18-4e8e-86e0-78b5b1258e31 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:44.669188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:44.694639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b6089f-4358-4856-9a56-e78fec451450 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:44.728933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68158c0-b43e-4054-b350-17447ad9b4d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:44.729456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68158c0-b43e-4054-b350-17447ad9b4d9 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:44.729730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68158c0-b43e-4054-b350-17447ad9b4d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:44.729999 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68158c0-b43e-4054-b350-17447ad9b4d9 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:44.730769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68158c0-b43e-4054-b350-17447ad9b4d9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:44.731098 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68158c0-b43e-4054-b350-17447ad9b4d9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:44.732455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68158c0-b43e-4054-b350-17447ad9b4d9 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:44.740866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 312/617] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:27:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:44.746808 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68158c0-b43e-4054-b350-17447ad9b4d9 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:44.837870 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e731cb9-2c18-4e8e-86e0-78b5b1258e31 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:44.843332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:44.877296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68158c0-b43e-4054-b350-17447ad9b4d9 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:44.912586 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c323314-97d3-4420-bf2a-e980e571798a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:44.913245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c323314-97d3-4420-bf2a-e980e571798a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:44.913528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c323314-97d3-4420-bf2a-e980e571798a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:44.913805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c323314-97d3-4420-bf2a-e980e571798a None admin] PATH_INFO: `/v3/users/nova` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:44.926670 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1613dc-7dea-440d-a4e9-455587084f5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:44.926862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c323314-97d3-4420-bf2a-e980e571798a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:44.927809 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1613dc-7dea-440d-a4e9-455587084f5a None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:44.928308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1613dc-7dea-440d-a4e9-455587084f5a None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:44.928509 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c323314-97d3-4420-bf2a-e980e571798a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:44.928992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1613dc-7dea-440d-a4e9-455587084f5a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:44.941167 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1613dc-7dea-440d-a4e9-455587084f5a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:44.942984 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1613dc-7dea-440d-a4e9-455587084f5a None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:44.964232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-8c323314-97d3-4420-bf2a-e980e571798a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 16 03:27:44.964232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:44.964232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:44.964232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:44.964232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:44.964232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:44.964232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:44.964232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:44.964232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:44.964232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:44.964232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:44.964232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:44.964232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:27:44.964232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:27:44.964232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:27:44.964232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:27:44.965429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:44.965429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:44.965429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:27:44.965429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:44.965429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:27:44.965429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:44.965429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:44.965429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:44.965429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:44.965429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:44.965429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:44.965429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:44.965429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:44.965429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:44.965429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:44.965429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:44.965429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:44.965429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:44.966493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:44.966493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:44.966493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:27:44.966493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:27:44.966493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:27:44.966493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:27:44.966493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:27:44.966493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:27:44.966493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 16 03:27:44.966493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:44.966493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 306/618] 10.222.0.49 () {56 vars in 1093 bytes} [Fri Sep 16 03:27:44 2022] GET /identity/v3/users/nova => generated 82 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:44.971020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-5c1613dc-7dea-440d-a4e9-455587084f5a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:44.971020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:44.971020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:44.971020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:44.971020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:44.971020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:44.971020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:44.971020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:44.971020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:44.971020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:44.971020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:44.971020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:44.971020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:44.971020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:44.971020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:44.971020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:44.971970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:44.971970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:44.971970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:44.971970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:44.971970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:44.971970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:44.971970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:44.971970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:44.971970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:44.971970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:44.971970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:44.971970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:44.971970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:44.971970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:44.971970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:44.971970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:44.971970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:44.973064 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:44.973064 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:44.973064 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:44.975993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 313/619] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:44.977714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c323314-97d3-4420-bf2a-e980e571798a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:44.981700 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1613dc-7dea-440d-a4e9-455587084f5a None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:45.102184 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:45.140922 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c323314-97d3-4420-bf2a-e980e571798a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:45.180156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab61961b-cc67-4614-8751-85b9229b481d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:45.180913 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab61961b-cc67-4614-8751-85b9229b481d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:45.181197 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab61961b-cc67-4614-8751-85b9229b481d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:45.181466 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab61961b-cc67-4614-8751-85b9229b481d None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:45.182188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab61961b-cc67-4614-8751-85b9229b481d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:45.182489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab61961b-cc67-4614-8751-85b9229b481d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:45.183915 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab61961b-cc67-4614-8751-85b9229b481d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:45.184184 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1613dc-7dea-440d-a4e9-455587084f5a None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:45.219955 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 307/620] 10.222.0.49 () {56 vars in 1133 bytes} [Fri Sep 16 03:27:44 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:45.225551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab61961b-cc67-4614-8751-85b9229b481d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:45.231425 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:45.251782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73005200-baf0-4a8f-b325-181f33b3c42e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:45.252315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73005200-baf0-4a8f-b325-181f33b3c42e None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:45.252588 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73005200-baf0-4a8f-b325-181f33b3c42e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:45.252853 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73005200-baf0-4a8f-b325-181f33b3c42e None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:45.253488 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73005200-baf0-4a8f-b325-181f33b3c42e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:45.253781 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73005200-baf0-4a8f-b325-181f33b3c42e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:45.255178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73005200-baf0-4a8f-b325-181f33b3c42e None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:45.283225 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 314/621] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:45.290039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73005200-baf0-4a8f-b325-181f33b3c42e None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:45.416846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab61961b-cc67-4614-8751-85b9229b481d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:45.460380 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:45.462048 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73005200-baf0-4a8f-b325-181f33b3c42e None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:45.479104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07708caf-e564-4de6-9b82-28ee42732da6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:45.479731 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07708caf-e564-4de6-9b82-28ee42732da6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:45.480016 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07708caf-e564-4de6-9b82-28ee42732da6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:45.480293 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07708caf-e564-4de6-9b82-28ee42732da6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:45.498944 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa74195-449a-46f0-a39f-4b43f29fcfca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:45.499980 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa74195-449a-46f0-a39f-4b43f29fcfca None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:45.500998 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa74195-449a-46f0-a39f-4b43f29fcfca None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:45.502472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa74195-449a-46f0-a39f-4b43f29fcfca None admin] PATH_INFO: `/v3/users/swift` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:45.510363 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa74195-449a-46f0-a39f-4b43f29fcfca None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:45.512128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa74195-449a-46f0-a39f-4b43f29fcfca None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:45.524694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-1aa74195-449a-46f0-a39f-4b43f29fcfca None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 16 03:27:45.524694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:45.524694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:45.524694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:45.524694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:45.524694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:45.524694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:45.524694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:45.524694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:45.524694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:45.524694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:45.524694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:45.524694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:27:45.524694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:27:45.524694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:27:45.524694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:27:45.525681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:45.525681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:45.525681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:27:45.525681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:45.525681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:27:45.525681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:45.525681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:45.525681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:45.525681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:45.525681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:45.525681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:45.525681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:45.525681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:45.525681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:45.525681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:45.525681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:45.525681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:45.525681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:45.526621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:45.526621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:45.526621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:27:45.526621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:27:45.526621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:27:45.526621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:27:45.526621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:27:45.526621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:27:45.526621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 16 03:27:45.526621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:45.530481 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 315/622] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:27:45 2022] GET /identity/v3/users/swift => generated 83 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:45.537296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa74195-449a-46f0-a39f-4b43f29fcfca None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:45.586239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:45.604611 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07708caf-e564-4de6-9b82-28ee42732da6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:45.608767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07708caf-e564-4de6-9b82-28ee42732da6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:45.644366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa74195-449a-46f0-a39f-4b43f29fcfca None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:45.682592 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe033b4-1b16-4605-aa2e-4293a90938a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:45.683620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe033b4-1b16-4605-aa2e-4293a90938a8 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:45.684119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe033b4-1b16-4605-aa2e-4293a90938a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:45.684614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe033b4-1b16-4605-aa2e-4293a90938a8 None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:45.685600 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe033b4-1b16-4605-aa2e-4293a90938a8 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:45.686145 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe033b4-1b16-4605-aa2e-4293a90938a8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:45.688626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe033b4-1b16-4605-aa2e-4293a90938a8 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:45.707705 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 316/623] 10.222.0.49 () {56 vars in 1135 bytes} [Fri Sep 16 03:27:45 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:45.709289 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306f009c-1ae4-47d9-bdc6-4f1470b1a8aa None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:45.709784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306f009c-1ae4-47d9-bdc6-4f1470b1a8aa None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:45.710283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306f009c-1ae4-47d9-bdc6-4f1470b1a8aa None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:45.711956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 317/624] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:45.724063 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306f009c-1ae4-47d9-bdc6-4f1470b1a8aa None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:45.729157 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 308/625] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 500 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 16 03:27:45.730624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed2cd98-1923-4d44-94ca-1023e37e2b23 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:45.730932 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed2cd98-1923-4d44-94ca-1023e37e2b23 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:45.731220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed2cd98-1923-4d44-94ca-1023e37e2b23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:45.751645 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-3ed2cd98-1923-4d44-94ca-1023e37e2b23 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:45.752896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3ed2cd98-1923-4d44-94ca-1023e37e2b23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:45.773911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:45.831428 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 309/626] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:45.832411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed2cd98-1923-4d44-94ca-1023e37e2b23 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:45.877356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306f009c-1ae4-47d9-bdc6-4f1470b1a8aa None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:45.881276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:45.944826 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0cf4c4-0d2f-471f-be9a-946c94d2c9b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:45.945490 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0cf4c4-0d2f-471f-be9a-946c94d2c9b1 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:45.945768 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0cf4c4-0d2f-471f-be9a-946c94d2c9b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:45.946076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0cf4c4-0d2f-471f-be9a-946c94d2c9b1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:45.973546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0cf4c4-0d2f-471f-be9a-946c94d2c9b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:45.975162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0cf4c4-0d2f-471f-be9a-946c94d2c9b1 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:46.002131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-ed0cf4c4-0d2f-471f-be9a-946c94d2c9b1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:46.002131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:46.002131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:46.002131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:46.002131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:46.002131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:46.002131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:46.002131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:46.002131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:46.002131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:46.002131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:46.002131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:46.002131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:46.002131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:46.002131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:46.002131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:46.009715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:46.009715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:46.009715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:46.009715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:46.009715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:46.009715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:46.009715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:46.009715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:46.009715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:46.009715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:46.009715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:46.009715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:46.009715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:46.009715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:46.009715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:46.009715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:46.009715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:46.010645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:46.010645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:46.010645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:46.010645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 318/627] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:46.010645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d79a9e-e79c-4ebd-903f-f911c4e0dfd6 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:46.010645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d79a9e-e79c-4ebd-903f-f911c4e0dfd6 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:46.010645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d79a9e-e79c-4ebd-903f-f911c4e0dfd6 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:46.014818 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 319/628] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:46 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:46.016737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d79a9e-e79c-4ebd-903f-f911c4e0dfd6 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:46.024204 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed2cd98-1923-4d44-94ca-1023e37e2b23 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:46.071229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb059dd-49e8-476a-9864-e55f07d5c06a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:46.072390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb059dd-49e8-476a-9864-e55f07d5c06a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:46.072875 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb059dd-49e8-476a-9864-e55f07d5c06a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:46.073302 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb059dd-49e8-476a-9864-e55f07d5c06a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:46.089661 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb059dd-49e8-476a-9864-e55f07d5c06a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:46.091918 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb059dd-49e8-476a-9864-e55f07d5c06a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:46.118047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-dfb059dd-49e8-476a-9864-e55f07d5c06a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:46.118047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:46.118047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:46.118047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:46.118047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:46.118047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:46.118047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:46.118047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:46.118047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:46.118047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:46.118047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:46.118047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:46.118047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:46.118047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:46.118047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:46.118047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:46.119261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:46.119261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:46.119261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:46.119261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:46.119261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:46.119261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:46.119261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:46.119261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:46.119261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:46.119261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:46.119261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:46.119261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:46.119261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:46.119261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:46.119261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:46.119261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:46.119261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:46.120468 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:46.120468 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:46.120468 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:46.127304 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 310/629] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:46.128303 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f73466a-6a71-460f-9f3e-b487f9c55a12 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:46.128587 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f73466a-6a71-460f-9f3e-b487f9c55a12 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:46.128864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f73466a-6a71-460f-9f3e-b487f9c55a12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:46.143831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:46.144846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d79a9e-e79c-4ebd-903f-f911c4e0dfd6 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:46.149625 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9f73466a-6a71-460f-9f3e-b487f9c55a12 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:46.150970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9f73466a-6a71-460f-9f3e-b487f9c55a12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:46.182260 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23aaf6fd-1ea7-4bcd-8608-7337667a490b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:46.183200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23aaf6fd-1ea7-4bcd-8608-7337667a490b None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:46.183486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23aaf6fd-1ea7-4bcd-8608-7337667a490b None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:46.183763 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23aaf6fd-1ea7-4bcd-8608-7337667a490b None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:46.184424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23aaf6fd-1ea7-4bcd-8608-7337667a490b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:46.184972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23aaf6fd-1ea7-4bcd-8608-7337667a490b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:46.186078 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23aaf6fd-1ea7-4bcd-8608-7337667a490b None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:46.198684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 320/630] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:46.199561 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23aaf6fd-1ea7-4bcd-8608-7337667a490b None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:46.214427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:46.233801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 311/631] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:46.233801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f73466a-6a71-460f-9f3e-b487f9c55a12 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:46.302596 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23aaf6fd-1ea7-4bcd-8608-7337667a490b None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:46.338793 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f73466a-6a71-460f-9f3e-b487f9c55a12 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:46.339814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2266fcf-7bc5-4851-8dc5-7a7a07590b62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:46.340537 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2266fcf-7bc5-4851-8dc5-7a7a07590b62 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:46.340814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2266fcf-7bc5-4851-8dc5-7a7a07590b62 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:46.341195 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2266fcf-7bc5-4851-8dc5-7a7a07590b62 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:46.341837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2266fcf-7bc5-4851-8dc5-7a7a07590b62 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:46.342306 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2266fcf-7bc5-4851-8dc5-7a7a07590b62 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:46.343949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2266fcf-7bc5-4851-8dc5-7a7a07590b62 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:46.362212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 321/632] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:46.362212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9528558e-388d-4913-886d-4d522e0568c1 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:46.362212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9528558e-388d-4913-886d-4d522e0568c1 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:46.362212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9528558e-388d-4913-886d-4d522e0568c1 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:46.362212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 322/633] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:46.362212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9528558e-388d-4913-886d-4d522e0568c1 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:46.380641 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b8222c-f52a-4cde-b0a8-b9671bf5a642 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:46.381276 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b8222c-f52a-4cde-b0a8-b9671bf5a642 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:46.381558 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b8222c-f52a-4cde-b0a8-b9671bf5a642 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:46.381830 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b8222c-f52a-4cde-b0a8-b9671bf5a642 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:46.384732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:46.393943 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b8222c-f52a-4cde-b0a8-b9671bf5a642 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:46.398992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-e5b8222c-f52a-4cde-b0a8-b9671bf5a642 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:46.398992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:46.398992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:46.398992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:46.398992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:46.398992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:46.398992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:46.398992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:46.398992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:46.398992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:46.398992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:46.398992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:46.398992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:46.398992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:46.398992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:46.398992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:46.400324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:46.400324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:46.400324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:46.400324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:46.400324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:46.400324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:46.400324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:46.400324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:46.400324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:46.400324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:46.400324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:46.400324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:46.400324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:46.400324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:46.400324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:46.400324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:46.401188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:46.401188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:46.401188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:46.401188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:46.401188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:46.401188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:46.401188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:46.401188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:46.401188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:46.401188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:46.401188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:46.401188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:46.401188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 312/634] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:27:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:46.401188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b8222c-f52a-4cde-b0a8-b9671bf5a642 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:46.448830 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:46.479479 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9528558e-388d-4913-886d-4d522e0568c1 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:46.510232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b8222c-f52a-4cde-b0a8-b9671bf5a642 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:46.520143 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff9a778-f428-4a50-9543-b584bbf89d0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:46.520785 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff9a778-f428-4a50-9543-b584bbf89d0e None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:46.521069 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff9a778-f428-4a50-9543-b584bbf89d0e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:46.521345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff9a778-f428-4a50-9543-b584bbf89d0e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:46.528250 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff9a778-f428-4a50-9543-b584bbf89d0e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:46.529698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff9a778-f428-4a50-9543-b584bbf89d0e None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:46.541842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-aff9a778-f428-4a50-9543-b584bbf89d0e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:46.541842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:46.541842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:46.541842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:46.541842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:46.541842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:46.541842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:46.541842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:46.541842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:46.541842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:46.541842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:46.541842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:46.541842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:27:46.541842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:27:46.541842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:27:46.542754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:27:46.542754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:46.542754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:46.542754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:46.542754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:46.542754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:46.542754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:46.542754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:46.542754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:46.542754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:46.542754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:46.542754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:46.542754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:46.542754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:46.542754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:46.542754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:46.542754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:46.543710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:27:46.543710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:27:46.543710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:27:46.543710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:27:46.543710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:27:46.543710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:27:46.543710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:46.543710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:46.543710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 323/635] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:46.544348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681c17c5-0d11-49b5-8a0e-c89fc730baf7 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:46.544629 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681c17c5-0d11-49b5-8a0e-c89fc730baf7 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:46.544901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681c17c5-0d11-49b5-8a0e-c89fc730baf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:46.550320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b17b66-a181-4081-a6ff-fc14287c666c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:46.550992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b17b66-a181-4081-a6ff-fc14287c666c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:46.551283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b17b66-a181-4081-a6ff-fc14287c666c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:46.551560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b17b66-a181-4081-a6ff-fc14287c666c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:46.558472 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b17b66-a181-4081-a6ff-fc14287c666c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:46.559881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b17b66-a181-4081-a6ff-fc14287c666c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:46.562703 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-681c17c5-0d11-49b5-8a0e-c89fc730baf7 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:46.564321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-681c17c5-0d11-49b5-8a0e-c89fc730baf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:46.570988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-51b17b66-a181-4081-a6ff-fc14287c666c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:46.570988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:46.570988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:46.570988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:46.570988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:46.570988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:46.570988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:46.570988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:46.570988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:46.570988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:46.570988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:46.570988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:46.570988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:27:46.570988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:27:46.570988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:27:46.571928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:27:46.571928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:46.571928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:46.571928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:46.571928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:46.571928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:46.571928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:46.571928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:46.571928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:46.571928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:46.571928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:46.571928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:46.571928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:46.571928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:46.571928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:46.571928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:46.571928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:46.572837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:27:46.572837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:27:46.572837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:27:46.572837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:27:46.572837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:27:46.572837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:27:46.572837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:46.572837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:46.572837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 313/636] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:46.572837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b17b66-a181-4081-a6ff-fc14287c666c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:46.629232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:46.630517 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 324/637] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:46.631391 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681c17c5-0d11-49b5-8a0e-c89fc730baf7 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:46.682538 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b17b66-a181-4081-a6ff-fc14287c666c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:46.703966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:46.722103 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4124d0c0-37b3-4815-b17f-686311ce3ef8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:46.722714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4124d0c0-37b3-4815-b17f-686311ce3ef8 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:46.723025 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4124d0c0-37b3-4815-b17f-686311ce3ef8 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:46.723306 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4124d0c0-37b3-4815-b17f-686311ce3ef8 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:46.723957 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4124d0c0-37b3-4815-b17f-686311ce3ef8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:46.724365 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4124d0c0-37b3-4815-b17f-686311ce3ef8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:46.725719 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4124d0c0-37b3-4815-b17f-686311ce3ef8 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:46.741773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 314/638] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:27:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:46.741773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4124d0c0-37b3-4815-b17f-686311ce3ef8 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:46.747512 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681c17c5-0d11-49b5-8a0e-c89fc730baf7 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:46.784026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e443c55-1c4c-4e24-b7b4-0da96f9e198f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:46.784779 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e443c55-1c4c-4e24-b7b4-0da96f9e198f None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:46.785062 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e443c55-1c4c-4e24-b7b4-0da96f9e198f None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:46.785336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e443c55-1c4c-4e24-b7b4-0da96f9e198f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:46.795679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e443c55-1c4c-4e24-b7b4-0da96f9e198f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:46.801784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-0e443c55-1c4c-4e24-b7b4-0da96f9e198f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:46.801784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:46.801784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:46.801784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:46.801784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:46.801784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:46.801784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:46.801784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:46.801784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:46.801784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:46.801784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:46.801784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:46.801784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:46.801784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:46.801784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:46.801784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:46.803010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:46.803010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:46.803010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:46.803010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:46.803010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:46.803010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:46.803010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:46.803010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:46.803010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:46.803010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:46.803010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:46.803010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:46.803010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:46.803010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:46.803010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:46.803010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:46.804017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:46.804017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:46.804017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:46.804017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:46.804017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:46.804017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:46.804017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:46.804017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:46.804017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:46.804017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:46.804017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:46.804017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:46.804017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 325/639] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:27:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:46.804017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e443c55-1c4c-4e24-b7b4-0da96f9e198f None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:46.851331 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4124d0c0-37b3-4815-b17f-686311ce3ef8 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:46.869029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:46.885664 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84944def-6661-40f0-8ff5-a9a9b9afb332 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:46.886271 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84944def-6661-40f0-8ff5-a9a9b9afb332 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:46.886550 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84944def-6661-40f0-8ff5-a9a9b9afb332 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:46.886819 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84944def-6661-40f0-8ff5-a9a9b9afb332 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:46.887532 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84944def-6661-40f0-8ff5-a9a9b9afb332 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:46.887833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84944def-6661-40f0-8ff5-a9a9b9afb332 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:46.889237 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84944def-6661-40f0-8ff5-a9a9b9afb332 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:46.902501 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 315/640] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:27:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:46.903336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84944def-6661-40f0-8ff5-a9a9b9afb332 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:46.919199 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e443c55-1c4c-4e24-b7b4-0da96f9e198f None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:46.944129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:46.954821 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e772212-7fdc-4f63-bdea-7e7128a46a9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:46.955464 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e772212-7fdc-4f63-bdea-7e7128a46a9d None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:46.955743 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e772212-7fdc-4f63-bdea-7e7128a46a9d None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:46.956022 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e772212-7fdc-4f63-bdea-7e7128a46a9d None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:46.956704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e772212-7fdc-4f63-bdea-7e7128a46a9d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:46.957009 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e772212-7fdc-4f63-bdea-7e7128a46a9d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:46.958452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e772212-7fdc-4f63-bdea-7e7128a46a9d None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:46.975310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 326/641] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:27:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:46.975310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e772212-7fdc-4f63-bdea-7e7128a46a9d None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:47.009503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84944def-6661-40f0-8ff5-a9a9b9afb332 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:47.043626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec987fb3-4171-4bf4-9327-e54adda69183 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:47.044275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec987fb3-4171-4bf4-9327-e54adda69183 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:47.044528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec987fb3-4171-4bf4-9327-e54adda69183 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:47.044800 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec987fb3-4171-4bf4-9327-e54adda69183 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:47.055201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec987fb3-4171-4bf4-9327-e54adda69183 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:47.056621 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec987fb3-4171-4bf4-9327-e54adda69183 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:47.071954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-ec987fb3-4171-4bf4-9327-e54adda69183 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:47.071954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:47.071954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:47.071954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:47.071954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:47.071954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:47.071954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:47.071954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:47.071954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:47.071954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:47.071954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:47.071954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:47.071954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:47.071954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:47.071954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:47.071954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:47.073156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:47.073156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:47.073156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:47.073156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:47.073156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:47.073156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:47.073156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:47.073156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:47.073156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:47.073156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:47.073156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:47.073156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:47.073156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:47.073156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:47.073156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:47.073156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:47.073156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:47.074412 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:47.074412 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:47.074412 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:47.074412 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 316/642] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:47.074659 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e772212-7fdc-4f63-bdea-7e7128a46a9d None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:47.074735 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec987fb3-4171-4bf4-9327-e54adda69183 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:47.104092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:47.106820 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e0dced-04a4-468d-80e5-718cbb94d2db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:47.107457 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e0dced-04a4-468d-80e5-718cbb94d2db None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:47.107734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e0dced-04a4-468d-80e5-718cbb94d2db None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:47.108001 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e0dced-04a4-468d-80e5-718cbb94d2db None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:47.108646 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e0dced-04a4-468d-80e5-718cbb94d2db None admin] RBAC: Adding query filter params (name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:47.108945 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e0dced-04a4-468d-80e5-718cbb94d2db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:47.110859 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e0dced-04a4-468d-80e5-718cbb94d2db None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:47.118510 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 327/643] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:27:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:47.119289 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e0dced-04a4-468d-80e5-718cbb94d2db None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:47.166347 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:47.172229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec987fb3-4171-4bf4-9327-e54adda69183 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:47.206300 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd935851-da60-47a4-8529-47c83cf767bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:47.206974 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd935851-da60-47a4-8529-47c83cf767bb None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:47.207357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd935851-da60-47a4-8529-47c83cf767bb None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:47.207640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd935851-da60-47a4-8529-47c83cf767bb None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:47.217383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd935851-da60-47a4-8529-47c83cf767bb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:47.218797 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e0dced-04a4-468d-80e5-718cbb94d2db None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:47.222856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-bd935851-da60-47a4-8529-47c83cf767bb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:47.222856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:47.222856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:47.222856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:47.222856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:47.222856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:47.222856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:47.222856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:47.222856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:47.222856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:47.222856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:47.222856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:47.222856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:47.222856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:47.222856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:47.222856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:47.224000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:47.224000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:47.224000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:47.224000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:47.224000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:47.224000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:47.224000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:47.224000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:47.224000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:47.224000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:47.224000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:47.224000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:47.224000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:47.224000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:47.224000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:47.224000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:47.225104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:47.225104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:47.225104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:47.225104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:47.225104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:47.225104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:47.225104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:47.225104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:47.225104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:47.225104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:47.225104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:47.225104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:47.225104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 317/644] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:27:47 2022] GET /identity/v3/roles/service => generated 85 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:47.225104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd935851-da60-47a4-8529-47c83cf767bb None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:47.255867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233a6ed0-861a-45de-bd6a-889d1d438bc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:47.256524 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233a6ed0-861a-45de-bd6a-889d1d438bc5 None admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:47.256808 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233a6ed0-861a-45de-bd6a-889d1d438bc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:47.257086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233a6ed0-861a-45de-bd6a-889d1d438bc5 None admin] PATH_INFO: `/v3/projects/0bb7350474f349d6aee197ff35536167/users/aa30da9972e140d7905c522576681429/roles/974d839067e9411aa4cb850cee33f6ae` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:47.271758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233a6ed0-861a-45de-bd6a-889d1d438bc5 None admin] RBAC: Authorizing `identity:create_grant(project_id=0bb7350474f349d6aee197ff35536167, user_id=aa30da9972e140d7905c522576681429, role_id=974d839067e9411aa4cb850cee33f6ae)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:47.273222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233a6ed0-861a-45de-bd6a-889d1d438bc5 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:47.313758 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:47.322653 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd935851-da60-47a4-8529-47c83cf767bb None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:47.355555 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdab31ac-490c-41b9-a610-ba6562cf7589 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:47.356089 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdab31ac-490c-41b9-a610-ba6562cf7589 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:47.356365 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdab31ac-490c-41b9-a610-ba6562cf7589 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:47.356635 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdab31ac-490c-41b9-a610-ba6562cf7589 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:47.357340 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdab31ac-490c-41b9-a610-ba6562cf7589 None admin] RBAC: Adding query filter params (role.id=974d839067e9411aa4cb850cee33f6ae, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=ded1bcd10e62499a9aaa0ece5aa55c80) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:47.357646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdab31ac-490c-41b9-a610-ba6562cf7589 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:47.359086 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdab31ac-490c-41b9-a610-ba6562cf7589 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:47.364661 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 318/645] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:27:47 2022] GET /identity/v3/role_assignments?user.id=ded1bcd10e62499a9aaa0ece5aa55c80&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=974d839067e9411aa4cb850cee33f6ae => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:47.365409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdab31ac-490c-41b9-a610-ba6562cf7589 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:47.406206 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 328/646] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:27:47 2022] PUT /identity/v3/projects/0bb7350474f349d6aee197ff35536167/users/aa30da9972e140d7905c522576681429/roles/974d839067e9411aa4cb850cee33f6ae => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:27:47.408446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233a6ed0-861a-45de-bd6a-889d1d438bc5 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:47.510956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdab31ac-490c-41b9-a610-ba6562cf7589 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:47.538154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:47.546973 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233a6ed0-861a-45de-bd6a-889d1d438bc5 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:47.560764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d8e050-6600-4734-9ec5-bb195f51d89d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:47.561782 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d8e050-6600-4734-9ec5-bb195f51d89d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:47.562228 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d8e050-6600-4734-9ec5-bb195f51d89d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:47.562505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d8e050-6600-4734-9ec5-bb195f51d89d None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:47.563472 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d8e050-6600-4734-9ec5-bb195f51d89d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:47.563974 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d8e050-6600-4734-9ec5-bb195f51d89d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:47.565395 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d8e050-6600-4734-9ec5-bb195f51d89d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:47.583444 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 319/647] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:47.584213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d8e050-6600-4734-9ec5-bb195f51d89d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:47.602563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869ce44b-0e3f-4cf6-a619-928bae39b139 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:47.603989 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869ce44b-0e3f-4cf6-a619-928bae39b139 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:47.604454 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869ce44b-0e3f-4cf6-a619-928bae39b139 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:47.605049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869ce44b-0e3f-4cf6-a619-928bae39b139 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:47.610150 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:47.621130 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869ce44b-0e3f-4cf6-a619-928bae39b139 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:47.621130 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869ce44b-0e3f-4cf6-a619-928bae39b139 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:47.636958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-869ce44b-0e3f-4cf6-a619-928bae39b139 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:47.636958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:47.636958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:47.636958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:47.636958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:47.636958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:47.636958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:47.636958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:47.636958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:47.636958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:47.636958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:47.636958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:47.636958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:47.636958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:47.636958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:47.636958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:47.637952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:47.637952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:47.637952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:47.637952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:47.637952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:47.637952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:47.637952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:47.637952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:47.637952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:47.637952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:47.637952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:47.637952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:47.637952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:47.637952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:47.637952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:47.637952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:47.637952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:47.638853 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:47.638853 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:47.638853 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:47.638853 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 329/648] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:47.639146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869ce44b-0e3f-4cf6-a619-928bae39b139 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:47.714893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d8e050-6600-4734-9ec5-bb195f51d89d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:47.759170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599ca839-6bbd-4224-a7d3-923adb569bbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:47.760204 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599ca839-6bbd-4224-a7d3-923adb569bbb None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:47.760750 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599ca839-6bbd-4224-a7d3-923adb569bbb None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:47.761031 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599ca839-6bbd-4224-a7d3-923adb569bbb None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:47.761795 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599ca839-6bbd-4224-a7d3-923adb569bbb None admin] RBAC: Adding query filter params (name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:47.762101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599ca839-6bbd-4224-a7d3-923adb569bbb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:47.763631 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599ca839-6bbd-4224-a7d3-923adb569bbb None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:47.765547 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869ce44b-0e3f-4cf6-a619-928bae39b139 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:47.779506 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 320/649] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:27:47 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:47.780901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599ca839-6bbd-4224-a7d3-923adb569bbb None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:47.808931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19fb51e-d244-4ca7-8be5-52945705abc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:47.809897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19fb51e-d244-4ca7-8be5-52945705abc2 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:47.810186 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19fb51e-d244-4ca7-8be5-52945705abc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:47.810461 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19fb51e-d244-4ca7-8be5-52945705abc2 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:47.822757 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19fb51e-d244-4ca7-8be5-52945705abc2 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:47.824269 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19fb51e-d244-4ca7-8be5-52945705abc2 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:47.839615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-c19fb51e-d244-4ca7-8be5-52945705abc2 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 16 03:27:47.839615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:47.839615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:47.839615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:47.839615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:47.839615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:47.839615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:47.839615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:47.839615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:47.839615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:47.839615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:47.839615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:47.839615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:27:47.839615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:27:47.839615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:27:47.839615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:27:47.841745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:47.841745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:47.841745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:27:47.841745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:47.841745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:27:47.841745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:47.841745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:47.841745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:47.841745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:47.841745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:47.841745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:47.841745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:47.841745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:47.841745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:47.841745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:47.841745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:47.841745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:47.841745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:47.843953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:47.843953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:47.843953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:27:47.843953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:27:47.843953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:27:47.843953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:27:47.843953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:27:47.843953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:27:47.843953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 16 03:27:47.843953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:47.843953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 330/650] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:47 2022] GET /identity/v3/users/glance => generated 84 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:47.843953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19fb51e-d244-4ca7-8be5-52945705abc2 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:47.857197 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:47.872854 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:47.901838 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599ca839-6bbd-4224-a7d3-923adb569bbb None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:47.944934 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae28414-a4bf-42ff-b391-b17b7db14d40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:47.945731 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae28414-a4bf-42ff-b391-b17b7db14d40 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:47.946093 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae28414-a4bf-42ff-b391-b17b7db14d40 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:47.946444 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae28414-a4bf-42ff-b391-b17b7db14d40 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:47.947196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae28414-a4bf-42ff-b391-b17b7db14d40 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:47.947856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae28414-a4bf-42ff-b391-b17b7db14d40 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:47.949746 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae28414-a4bf-42ff-b391-b17b7db14d40 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:47.968454 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19fb51e-d244-4ca7-8be5-52945705abc2 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:47.973799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 321/651] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:47.974878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae28414-a4bf-42ff-b391-b17b7db14d40 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:48.019099 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f65b8b4-134f-4f2e-bed6-3a154b71734e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:48.020659 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f65b8b4-134f-4f2e-bed6-3a154b71734e None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:48.020937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f65b8b4-134f-4f2e-bed6-3a154b71734e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:48.021208 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f65b8b4-134f-4f2e-bed6-3a154b71734e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:48.037019 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f65b8b4-134f-4f2e-bed6-3a154b71734e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:48.038485 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f65b8b4-134f-4f2e-bed6-3a154b71734e None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:48.058351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-5f65b8b4-134f-4f2e-bed6-3a154b71734e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:48.058351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:48.058351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:48.058351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:48.058351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:48.058351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:48.058351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:48.058351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:48.058351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:48.058351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:48.058351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:48.058351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:48.058351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:48.058351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:48.058351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:48.058351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:48.059377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:48.059377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:48.059377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:48.059377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:48.059377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:48.059377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:48.059377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:48.059377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:48.059377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:48.059377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:48.059377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:48.059377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:48.059377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:48.059377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:48.059377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:48.059377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:48.059377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:48.060291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:48.060291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:48.060291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:48.060864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 331/652] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:48.061825 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f65b8b4-134f-4f2e-bed6-3a154b71734e None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:48.106158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae28414-a4bf-42ff-b391-b17b7db14d40 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:48.134871 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:48.153527 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236086d8-98e6-4c4f-b304-70bed60b9dfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:48.155138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236086d8-98e6-4c4f-b304-70bed60b9dfc None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:48.155138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236086d8-98e6-4c4f-b304-70bed60b9dfc None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:48.155428 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236086d8-98e6-4c4f-b304-70bed60b9dfc None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:48.156317 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236086d8-98e6-4c4f-b304-70bed60b9dfc None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:48.156687 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236086d8-98e6-4c4f-b304-70bed60b9dfc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:48.158678 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236086d8-98e6-4c4f-b304-70bed60b9dfc None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:48.162110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:48.181330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f65b8b4-134f-4f2e-bed6-3a154b71734e None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:48.181982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 322/653] 10.222.0.49 () {56 vars in 1137 bytes} [Fri Sep 16 03:27:47 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:48.183107 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236086d8-98e6-4c4f-b304-70bed60b9dfc None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:48.232596 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f459e7-9e47-427b-9831-15be9edbfd5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:48.233607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f459e7-9e47-427b-9831-15be9edbfd5b None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:48.233904 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f459e7-9e47-427b-9831-15be9edbfd5b None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:48.234177 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f459e7-9e47-427b-9831-15be9edbfd5b None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:48.244036 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f459e7-9e47-427b-9831-15be9edbfd5b None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:48.245549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f459e7-9e47-427b-9831-15be9edbfd5b None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:48.257996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-d6f459e7-9e47-427b-9831-15be9edbfd5b None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 16 03:27:48.257996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:48.257996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:48.257996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:48.257996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:48.257996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:48.257996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:48.257996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:48.257996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:48.257996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:48.257996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:48.257996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:48.257996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:27:48.257996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:27:48.257996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:27:48.257996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:27:48.260222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:48.260222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:48.260222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:27:48.260222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:48.260222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:27:48.260222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:48.260222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:48.260222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:48.260222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:48.260222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:48.260222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:48.260222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:48.260222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:48.260222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:48.260222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:48.260222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:48.260222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:48.260222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:48.262445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:48.262445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:48.262445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:27:48.262445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:27:48.262445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:27:48.262445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:27:48.262445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:27:48.262445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:27:48.262445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 16 03:27:48.262445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:48.267585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 332/654] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:27:48 2022] GET /identity/v3/users/neutron => generated 85 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:48.267585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f459e7-9e47-427b-9831-15be9edbfd5b None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:48.310035 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236086d8-98e6-4c4f-b304-70bed60b9dfc None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:48.357602 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d062973-67b5-47bb-8b65-e5f6c7286f36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:48.358214 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d062973-67b5-47bb-8b65-e5f6c7286f36 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:48.358489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d062973-67b5-47bb-8b65-e5f6c7286f36 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:48.358759 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d062973-67b5-47bb-8b65-e5f6c7286f36 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:48.359431 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d062973-67b5-47bb-8b65-e5f6c7286f36 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:48.360293 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d062973-67b5-47bb-8b65-e5f6c7286f36 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:48.361756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d062973-67b5-47bb-8b65-e5f6c7286f36 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:48.379193 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 323/655] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:48.379193 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d062973-67b5-47bb-8b65-e5f6c7286f36 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:48.393570 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f459e7-9e47-427b-9831-15be9edbfd5b None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:48.415700 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:48.431356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01172cf5-4bc3-4ac9-8b15-7517576114fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:48.432391 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01172cf5-4bc3-4ac9-8b15-7517576114fe None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:48.432826 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01172cf5-4bc3-4ac9-8b15-7517576114fe None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:48.433268 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01172cf5-4bc3-4ac9-8b15-7517576114fe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:48.440593 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:48.492282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d062973-67b5-47bb-8b65-e5f6c7286f36 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:48.511692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01172cf5-4bc3-4ac9-8b15-7517576114fe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:48.513438 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01172cf5-4bc3-4ac9-8b15-7517576114fe None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:48.532675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a50e7c-7219-4625-bc71-33f3742500d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:48.533278 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a50e7c-7219-4625-bc71-33f3742500d9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:48.533812 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a50e7c-7219-4625-bc71-33f3742500d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:48.534082 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a50e7c-7219-4625-bc71-33f3742500d9 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:48.535039 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a50e7c-7219-4625-bc71-33f3742500d9 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:48.538040 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a50e7c-7219-4625-bc71-33f3742500d9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:48.539649 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a50e7c-7219-4625-bc71-33f3742500d9 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:48.563782 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 324/656] 10.222.0.49 () {56 vars in 1139 bytes} [Fri Sep 16 03:27:48 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:48.564935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a50e7c-7219-4625-bc71-33f3742500d9 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:48.643007 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 333/657] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:27:48.643952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01172cf5-4bc3-4ac9-8b15-7517576114fe None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:48.674968 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:48.686183 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a50e7c-7219-4625-bc71-33f3742500d9 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:48.702866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:48.728543 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b00437-ffad-4789-ba76-a119690bead4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:48.729284 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b00437-ffad-4789-ba76-a119690bead4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:48.729688 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b00437-ffad-4789-ba76-a119690bead4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:48.730066 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b00437-ffad-4789-ba76-a119690bead4 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:48.738554 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b00437-ffad-4789-ba76-a119690bead4 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:48.740129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b00437-ffad-4789-ba76-a119690bead4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:48.753873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-61b00437-ffad-4789-ba76-a119690bead4 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 16 03:27:48.753873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:48.753873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:48.753873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:48.753873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:48.753873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:48.753873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:48.753873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:48.753873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:48.753873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:48.753873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:48.753873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:48.753873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:27:48.753873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:27:48.753873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:27:48.753873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:27:48.754871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:48.754871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:48.754871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:27:48.754871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:48.754871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:27:48.754871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:48.754871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:48.754871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:48.754871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:48.754871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:48.754871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:48.754871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:48.754871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:48.754871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:48.754871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:48.754871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:48.754871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:48.754871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:48.755849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:48.755849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:48.755849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:27:48.755849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:27:48.755849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:27:48.755849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:27:48.755849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:27:48.755849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:27:48.755849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 16 03:27:48.755849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:48.759054 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 325/658] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:48 2022] GET /identity/v3/users/cinder => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:48.760080 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b00437-ffad-4789-ba76-a119690bead4 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:48.780964 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01172cf5-4bc3-4ac9-8b15-7517576114fe None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:48.825480 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fc6cb5-8d4e-47b6-8bb1-fe46ae6dc6a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:48.826096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fc6cb5-8d4e-47b6-8bb1-fe46ae6dc6a1 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:48.826624 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fc6cb5-8d4e-47b6-8bb1-fe46ae6dc6a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:48.826898 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fc6cb5-8d4e-47b6-8bb1-fe46ae6dc6a1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:48.867398 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b00437-ffad-4789-ba76-a119690bead4 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:48.902382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237125ce-d05b-4cb6-b06f-7e9ca115adb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:48.902959 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237125ce-d05b-4cb6-b06f-7e9ca115adb2 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:48.903241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237125ce-d05b-4cb6-b06f-7e9ca115adb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:48.903513 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237125ce-d05b-4cb6-b06f-7e9ca115adb2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:48.917630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237125ce-d05b-4cb6-b06f-7e9ca115adb2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:48.919075 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237125ce-d05b-4cb6-b06f-7e9ca115adb2 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:48.919655 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fc6cb5-8d4e-47b6-8bb1-fe46ae6dc6a1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:48.921772 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fc6cb5-8d4e-47b6-8bb1-fe46ae6dc6a1 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:48.933834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-237125ce-d05b-4cb6-b06f-7e9ca115adb2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:48.933834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:48.933834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:48.933834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:48.933834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:48.933834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:48.933834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:48.933834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:48.933834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:48.933834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:48.933834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:48.933834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:48.933834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:48.933834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:48.933834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:48.933834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:48.934801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:48.934801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:48.934801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:48.934801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:48.934801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:48.934801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:48.934801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:48.934801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:48.934801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:48.934801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:48.934801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:48.934801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:48.934801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:48.934801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:48.934801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:48.934801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:48.934801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:48.935763 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:48.935763 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:48.935763 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:48.935763 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 326/659] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:48.935763 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237125ce-d05b-4cb6-b06f-7e9ca115adb2 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:48.946633 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:48.968420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:49.045305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 334/660] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:27:49.046199 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fc6cb5-8d4e-47b6-8bb1-fe46ae6dc6a1 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:49.063505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237125ce-d05b-4cb6-b06f-7e9ca115adb2 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:49.112085 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddb54fb-628b-44fe-a383-cf86265f0787 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:49.112951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddb54fb-628b-44fe-a383-cf86265f0787 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:49.113379 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddb54fb-628b-44fe-a383-cf86265f0787 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:49.114023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddb54fb-628b-44fe-a383-cf86265f0787 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:49.115388 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddb54fb-628b-44fe-a383-cf86265f0787 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:49.115699 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddb54fb-628b-44fe-a383-cf86265f0787 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:49.117123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddb54fb-628b-44fe-a383-cf86265f0787 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:49.136527 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 327/661] 10.222.0.49 () {56 vars in 1137 bytes} [Fri Sep 16 03:27:48 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:49.137544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddb54fb-628b-44fe-a383-cf86265f0787 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:49.165126 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fc6cb5-8d4e-47b6-8bb1-fe46ae6dc6a1 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:49.193232 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:49.214490 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abef7ce-623e-4bf0-bb53-8e1aec0facac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:49.215415 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abef7ce-623e-4bf0-bb53-8e1aec0facac None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:49.215868 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abef7ce-623e-4bf0-bb53-8e1aec0facac None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:49.216300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abef7ce-623e-4bf0-bb53-8e1aec0facac None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:49.217095 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abef7ce-623e-4bf0-bb53-8e1aec0facac None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:49.217542 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abef7ce-623e-4bf0-bb53-8e1aec0facac None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:49.220146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abef7ce-623e-4bf0-bb53-8e1aec0facac None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:49.226419 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:49.237777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 335/662] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:49.239192 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abef7ce-623e-4bf0-bb53-8e1aec0facac None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:49.250077 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddb54fb-628b-44fe-a383-cf86265f0787 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:49.288770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f548451-d9ed-4e3f-a1b7-f672f72b2400 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:49.289641 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f548451-d9ed-4e3f-a1b7-f672f72b2400 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:49.290015 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f548451-d9ed-4e3f-a1b7-f672f72b2400 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:49.290386 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f548451-d9ed-4e3f-a1b7-f672f72b2400 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:49.301394 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f548451-d9ed-4e3f-a1b7-f672f72b2400 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:49.303053 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f548451-d9ed-4e3f-a1b7-f672f72b2400 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:49.321864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-7f548451-d9ed-4e3f-a1b7-f672f72b2400 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:49.321864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:49.321864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:49.321864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:49.321864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:49.321864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:49.321864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:49.321864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:49.321864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:49.321864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:49.321864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:49.321864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:49.321864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:49.321864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:49.321864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:49.321864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:49.323123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:49.323123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:49.323123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:49.323123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:49.323123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:49.323123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:49.323123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:49.323123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:49.323123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:49.323123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:49.323123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:49.323123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:49.323123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:49.323123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:49.323123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:49.323123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:49.323123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:49.324198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:49.324198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:49.324198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:49.328228 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 328/663] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:49.328973 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f548451-d9ed-4e3f-a1b7-f672f72b2400 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:49.371676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abef7ce-623e-4bf0-bb53-8e1aec0facac None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:49.423453 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462ae068-8621-48f4-8865-acecab48c2b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:49.424299 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462ae068-8621-48f4-8865-acecab48c2b5 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:49.424746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462ae068-8621-48f4-8865-acecab48c2b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:49.425164 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462ae068-8621-48f4-8865-acecab48c2b5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:49.454026 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f548451-d9ed-4e3f-a1b7-f672f72b2400 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:49.493680 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:49.505762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea116a9-7f01-4d72-bd90-617fd1f58feb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:49.507187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea116a9-7f01-4d72-bd90-617fd1f58feb None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:49.507647 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea116a9-7f01-4d72-bd90-617fd1f58feb None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:49.508773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea116a9-7f01-4d72-bd90-617fd1f58feb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:49.511500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:49.517079 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462ae068-8621-48f4-8865-acecab48c2b5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:49.518750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462ae068-8621-48f4-8865-acecab48c2b5 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:49.520484 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea116a9-7f01-4d72-bd90-617fd1f58feb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:49.523265 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea116a9-7f01-4d72-bd90-617fd1f58feb None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:49.545177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-6ea116a9-7f01-4d72-bd90-617fd1f58feb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:49.545177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:49.545177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:49.545177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:49.545177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:49.545177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:49.545177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:49.545177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:49.545177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:49.545177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:49.545177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:49.545177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:49.545177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:27:49.545177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:27:49.545177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:27:49.546574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:27:49.546574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:49.546574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:49.546574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:49.546574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:49.546574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:49.546574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:49.546574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:49.546574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:49.546574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:49.546574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:49.546574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:49.546574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:49.546574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:49.546574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:49.546574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:49.546574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:49.548579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:27:49.548579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:27:49.548579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:27:49.548579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:27:49.548579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:27:49.548579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:27:49.548579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:49.548579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:49.549593 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 329/664] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:49.551077 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea116a9-7f01-4d72-bd90-617fd1f58feb None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:49.651447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 336/665] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:27:49.652283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462ae068-8621-48f4-8865-acecab48c2b5 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:49.684325 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea116a9-7f01-4d72-bd90-617fd1f58feb None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:49.722003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5833e0-3198-4074-b6a3-b01a24918aef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:49.722614 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5833e0-3198-4074-b6a3-b01a24918aef None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:49.722893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5833e0-3198-4074-b6a3-b01a24918aef None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:49.723194 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5833e0-3198-4074-b6a3-b01a24918aef None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:49.723879 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5833e0-3198-4074-b6a3-b01a24918aef None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:49.724201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5833e0-3198-4074-b6a3-b01a24918aef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:49.725535 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5833e0-3198-4074-b6a3-b01a24918aef None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:49.738762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 330/666] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:49.741157 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad38f51c-b6e9-42e5-bb84-7983db610a38 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:49.741684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad38f51c-b6e9-42e5-bb84-7983db610a38 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:49.742376 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad38f51c-b6e9-42e5-bb84-7983db610a38 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:49.744004 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:49.744421 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 331/667] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:49 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:49.746612 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad38f51c-b6e9-42e5-bb84-7983db610a38 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:49.764048 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462ae068-8621-48f4-8865-acecab48c2b5 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:49.800810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7cb448-bd18-4a13-98da-c18c5a87941b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:49.801418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7cb448-bd18-4a13-98da-c18c5a87941b None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:49.801694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7cb448-bd18-4a13-98da-c18c5a87941b None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:49.801964 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7cb448-bd18-4a13-98da-c18c5a87941b None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:49.802649 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7cb448-bd18-4a13-98da-c18c5a87941b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:49.802982 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7cb448-bd18-4a13-98da-c18c5a87941b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:49.804421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7cb448-bd18-4a13-98da-c18c5a87941b None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:49.808766 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:49.821390 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 337/668] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:27:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:49.821390 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7cb448-bd18-4a13-98da-c18c5a87941b None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:49.852987 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad38f51c-b6e9-42e5-bb84-7983db610a38 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:49.887342 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae9b366-a70b-4c65-bb9b-bda7bc3f6297 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:49.887960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae9b366-a70b-4c65-bb9b-bda7bc3f6297 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:49.888238 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae9b366-a70b-4c65-bb9b-bda7bc3f6297 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:49.888507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae9b366-a70b-4c65-bb9b-bda7bc3f6297 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:49.898485 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae9b366-a70b-4c65-bb9b-bda7bc3f6297 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:49.899966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae9b366-a70b-4c65-bb9b-bda7bc3f6297 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:49.914442 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7cb448-bd18-4a13-98da-c18c5a87941b None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:49.918696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-9ae9b366-a70b-4c65-bb9b-bda7bc3f6297 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:49.918696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:49.918696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:49.918696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:49.918696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:49.918696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:49.918696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:49.918696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:49.918696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:49.918696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:49.918696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:49.918696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:49.918696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:49.918696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:49.918696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:49.918696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:49.919829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:49.919829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:49.919829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:49.919829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:49.919829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:49.919829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:49.919829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:49.919829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:49.919829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:49.919829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:49.919829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:49.919829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:49.919829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:49.919829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:49.919829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:49.919829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:49.919829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:49.926123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:49.926123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:49.926123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:49.926123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 332/669] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:49.926123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52e330e-c093-4b87-97d5-2785c1472211 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:49.926123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52e330e-c093-4b87-97d5-2785c1472211 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:49.926123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52e330e-c093-4b87-97d5-2785c1472211 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:49.940762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-b52e330e-c093-4b87-97d5-2785c1472211 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:49.941900 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b52e330e-c093-4b87-97d5-2785c1472211 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:49.950549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03cfadab-566c-4a6e-aadb-167f4f08c0a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:49.951270 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cfadab-566c-4a6e-aadb-167f4f08c0a4 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:49.951550 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cfadab-566c-4a6e-aadb-167f4f08c0a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:49.951821 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cfadab-566c-4a6e-aadb-167f4f08c0a4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:49.959046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03cfadab-566c-4a6e-aadb-167f4f08c0a4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:49.960504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03cfadab-566c-4a6e-aadb-167f4f08c0a4 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:49.963187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:49.973619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-03cfadab-566c-4a6e-aadb-167f4f08c0a4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:49.973619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:49.973619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:49.973619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:49.973619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:49.973619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:49.973619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:49.973619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:49.973619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:49.973619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:49.973619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:49.973619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:49.973619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:27:49.973619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:27:49.973619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:27:49.974665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:27:49.974665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:49.974665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:49.974665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:49.974665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:49.974665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:49.974665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:49.974665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:49.974665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:49.974665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:49.974665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:49.974665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:49.974665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:49.974665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:49.974665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:49.974665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:49.974665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:49.976035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:27:49.976035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:27:49.976035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:27:49.976035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:27:49.976035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:27:49.976035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:27:49.976035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:49.976035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:49.976035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 338/670] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:49.976536 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67c0e3b-4cdd-4589-be50-b5050d70eb27 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:49.976604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67c0e3b-4cdd-4589-be50-b5050d70eb27 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:49.976826 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67c0e3b-4cdd-4589-be50-b5050d70eb27 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:49.978051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 339/671] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:49 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:49.978734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67c0e3b-4cdd-4589-be50-b5050d70eb27 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:50.014166 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 333/672] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:50.015055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52e330e-c093-4b87-97d5-2785c1472211 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:50.063381 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:50.093700 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67c0e3b-4cdd-4589-be50-b5050d70eb27 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:50.127159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b4f3df-3cf9-4e6e-a1ea-c876f85fb95b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:50.127754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b4f3df-3cf9-4e6e-a1ea-c876f85fb95b None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:50.128030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b4f3df-3cf9-4e6e-a1ea-c876f85fb95b None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:50.128309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b4f3df-3cf9-4e6e-a1ea-c876f85fb95b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:50.129002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b4f3df-3cf9-4e6e-a1ea-c876f85fb95b None admin] RBAC: Adding query filter params (role.id=974d839067e9411aa4cb850cee33f6ae, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=05ceb9fc31c345ccabaee8f30a3982a8) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:50.129311 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b4f3df-3cf9-4e6e-a1ea-c876f85fb95b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:50.130731 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b4f3df-3cf9-4e6e-a1ea-c876f85fb95b None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:50.136353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 340/673] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:27:49 2022] GET /identity/v3/role_assignments?user.id=05ceb9fc31c345ccabaee8f30a3982a8&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=974d839067e9411aa4cb850cee33f6ae => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:50.137119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b4f3df-3cf9-4e6e-a1ea-c876f85fb95b None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:50.137319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52e330e-c093-4b87-97d5-2785c1472211 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:50.186342 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f915bf9-7e83-4ada-bf6a-504e2236c973 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:50.187449 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f915bf9-7e83-4ada-bf6a-504e2236c973 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:50.187449 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f915bf9-7e83-4ada-bf6a-504e2236c973 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:50.188278 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f915bf9-7e83-4ada-bf6a-504e2236c973 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:50.188871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f915bf9-7e83-4ada-bf6a-504e2236c973 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:50.189353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f915bf9-7e83-4ada-bf6a-504e2236c973 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:50.190821 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f915bf9-7e83-4ada-bf6a-504e2236c973 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:50.204725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 334/674] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:50.205909 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24e717f-0058-4921-a835-d1f3253faa92 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:50.206190 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24e717f-0058-4921-a835-d1f3253faa92 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:50.206536 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24e717f-0058-4921-a835-d1f3253faa92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:50.216459 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:50.226633 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-b24e717f-0058-4921-a835-d1f3253faa92 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:50.227998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b24e717f-0058-4921-a835-d1f3253faa92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:50.272682 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b4f3df-3cf9-4e6e-a1ea-c876f85fb95b None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:50.318631 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 335/675] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:50.319855 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24e717f-0058-4921-a835-d1f3253faa92 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:50.324044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b1cc4a-d557-410b-a5d6-c6a4684662bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:50.325738 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b1cc4a-d557-410b-a5d6-c6a4684662bb None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:50.325738 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b1cc4a-d557-410b-a5d6-c6a4684662bb None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:50.325738 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b1cc4a-d557-410b-a5d6-c6a4684662bb None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:50.326287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b1cc4a-d557-410b-a5d6-c6a4684662bb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:50.326659 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b1cc4a-d557-410b-a5d6-c6a4684662bb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:50.328215 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b1cc4a-d557-410b-a5d6-c6a4684662bb None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:50.348542 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 341/676] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:27:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:50.349446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b1cc4a-d557-410b-a5d6-c6a4684662bb None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:50.353858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:50.450685 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24e717f-0058-4921-a835-d1f3253faa92 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:50.480638 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b1cc4a-d557-410b-a5d6-c6a4684662bb None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:50.498060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc33a715-d5e7-4c46-89f9-257f19e76fe8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:50.498813 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc33a715-d5e7-4c46-89f9-257f19e76fe8 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:50.499131 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc33a715-d5e7-4c46-89f9-257f19e76fe8 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:50.499403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc33a715-d5e7-4c46-89f9-257f19e76fe8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:50.508019 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc33a715-d5e7-4c46-89f9-257f19e76fe8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:50.509689 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc33a715-d5e7-4c46-89f9-257f19e76fe8 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:50.517367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:50.522788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-bc33a715-d5e7-4c46-89f9-257f19e76fe8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:50.522788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:50.522788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:50.522788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:50.522788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:50.522788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:50.522788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:50.522788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:50.522788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:50.522788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:50.522788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:50.522788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:50.522788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:27:50.522788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:27:50.522788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:27:50.523707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:27:50.523707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:50.523707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:50.523707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:50.523707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:50.523707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:50.523707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:50.523707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:50.523707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:50.523707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:50.523707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:50.523707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:50.523707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:50.523707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:50.523707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:50.523707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:50.523707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:50.524632 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:27:50.524632 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:27:50.524632 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:27:50.524632 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:27:50.524632 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:27:50.524632 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:27:50.524632 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:50.524632 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:50.524632 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 336/677] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:50.528327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a8e850-579b-4ebb-a0a7-58a0c8482a42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:50.528967 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a8e850-579b-4ebb-a0a7-58a0c8482a42 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:50.529491 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a8e850-579b-4ebb-a0a7-58a0c8482a42 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:50.529613 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc33a715-d5e7-4c46-89f9-257f19e76fe8 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:50.529695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a8e850-579b-4ebb-a0a7-58a0c8482a42 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:50.543904 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a8e850-579b-4ebb-a0a7-58a0c8482a42 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:50.551161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-41a8e850-579b-4ebb-a0a7-58a0c8482a42 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:50.551161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:50.551161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:50.551161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:50.551161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:50.551161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:50.551161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:50.551161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:50.551161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:50.551161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:50.551161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:50.551161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:50.551161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:50.551161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:50.551161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:50.551161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:50.552554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:50.552554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:50.552554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:50.552554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:50.552554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:50.552554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:50.552554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:50.552554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:50.552554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:50.552554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:50.552554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:50.552554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:50.552554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:50.552554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:50.552554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:50.552554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:50.554076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:50.554076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:50.554076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:50.554076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:50.554076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:50.554076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:50.554076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:50.554076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:50.554076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:50.554076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:50.554076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:50.554076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:50.562942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 342/678] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:27:50 2022] GET /identity/v3/roles/service => generated 85 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:50.562942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a8e850-579b-4ebb-a0a7-58a0c8482a42 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:50.630303 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:50.644863 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc33a715-d5e7-4c46-89f9-257f19e76fe8 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:50.679558 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fa0213-94de-4aae-9d9d-0f06c0a83b8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:50.680163 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fa0213-94de-4aae-9d9d-0f06c0a83b8f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:50.680443 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fa0213-94de-4aae-9d9d-0f06c0a83b8f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:50.680717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fa0213-94de-4aae-9d9d-0f06c0a83b8f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:50.681437 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fa0213-94de-4aae-9d9d-0f06c0a83b8f None admin] RBAC: Adding query filter params (role.id=974d839067e9411aa4cb850cee33f6ae, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=f9275eeba471423e9fa2a31a6e08676d) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:50.681747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fa0213-94de-4aae-9d9d-0f06c0a83b8f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:50.683188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fa0213-94de-4aae-9d9d-0f06c0a83b8f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:50.698939 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a8e850-579b-4ebb-a0a7-58a0c8482a42 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:50.699153 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 337/679] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:27:50 2022] GET /identity/v3/role_assignments?user.id=f9275eeba471423e9fa2a31a6e08676d&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=974d839067e9411aa4cb850cee33f6ae => generated 608 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:50.699153 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fa0213-94de-4aae-9d9d-0f06c0a83b8f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:50.740529 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971682d2-7e04-4aaf-80c3-dab58b206309 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:50.741069 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971682d2-7e04-4aaf-80c3-dab58b206309 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:50.741346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971682d2-7e04-4aaf-80c3-dab58b206309 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:50.741615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971682d2-7e04-4aaf-80c3-dab58b206309 None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:50.742428 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971682d2-7e04-4aaf-80c3-dab58b206309 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:50.742734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971682d2-7e04-4aaf-80c3-dab58b206309 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:50.744354 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971682d2-7e04-4aaf-80c3-dab58b206309 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:50.758453 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 343/680] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:27:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:50.759455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971682d2-7e04-4aaf-80c3-dab58b206309 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:50.810937 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fa0213-94de-4aae-9d9d-0f06c0a83b8f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:50.823113 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:50.850943 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01119cc8-2eff-446d-9fdb-8e022d206e7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:50.851847 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01119cc8-2eff-446d-9fdb-8e022d206e7b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:50.852225 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01119cc8-2eff-446d-9fdb-8e022d206e7b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:50.852597 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01119cc8-2eff-446d-9fdb-8e022d206e7b None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:50.853379 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01119cc8-2eff-446d-9fdb-8e022d206e7b None admin] RBAC: Adding query filter params (name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:50.853981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01119cc8-2eff-446d-9fdb-8e022d206e7b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:50.856062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01119cc8-2eff-446d-9fdb-8e022d206e7b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:50.870851 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 338/681] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:27:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:50.872198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01119cc8-2eff-446d-9fdb-8e022d206e7b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:50.893253 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:50.897200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971682d2-7e04-4aaf-80c3-dab58b206309 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:50.936714 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b308621d-a6b3-4a8a-8c4a-aca95282a174 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:50.937641 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b308621d-a6b3-4a8a-8c4a-aca95282a174 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:50.938582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b308621d-a6b3-4a8a-8c4a-aca95282a174 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:50.938865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b308621d-a6b3-4a8a-8c4a-aca95282a174 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:50.951617 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b308621d-a6b3-4a8a-8c4a-aca95282a174 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:50.957502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-b308621d-a6b3-4a8a-8c4a-aca95282a174 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:27:50.957502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:50.957502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:50.957502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:50.957502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:50.957502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:50.957502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:50.957502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:50.957502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:50.957502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:50.957502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:50.957502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:50.957502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:50.957502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:50.957502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:50.957502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:50.958467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:50.958467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:50.958467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:50.958467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:50.958467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:50.958467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:50.958467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:50.958467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:50.958467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:50.958467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:50.958467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:50.958467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:50.958467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:50.958467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:50.958467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:50.958467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:50.959816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:50.959816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:50.959816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:50.959816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:50.959816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:50.959816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:50.959816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:50.959816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:50.959816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:50.959816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:50.959816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:27:50.959816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:50.959816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 344/682] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:27:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:50.960532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b308621d-a6b3-4a8a-8c4a-aca95282a174 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:51.012058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01119cc8-2eff-446d-9fdb-8e022d206e7b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:51.057700 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1668e69d-8472-4abd-8b88-5ecd9da8a5c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:51.058321 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1668e69d-8472-4abd-8b88-5ecd9da8a5c4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:51.058603 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1668e69d-8472-4abd-8b88-5ecd9da8a5c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:51.058876 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1668e69d-8472-4abd-8b88-5ecd9da8a5c4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:51.059788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1668e69d-8472-4abd-8b88-5ecd9da8a5c4 None admin] RBAC: Adding query filter params (role.id=974d839067e9411aa4cb850cee33f6ae, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=e6aeffd924734771bc5bf0d282c9052d) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:51.060030 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1668e69d-8472-4abd-8b88-5ecd9da8a5c4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:51.061600 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1668e69d-8472-4abd-8b88-5ecd9da8a5c4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:51.074932 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 339/683] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:27:50 2022] GET /identity/v3/role_assignments?user.id=e6aeffd924734771bc5bf0d282c9052d&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=974d839067e9411aa4cb850cee33f6ae => generated 608 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:51.074932 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1668e69d-8472-4abd-8b88-5ecd9da8a5c4 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:51.122971 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b308621d-a6b3-4a8a-8c4a-aca95282a174 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:51.130600 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:51.209012 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c93d88-db41-49ad-9c97-842f304e81ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:51.209704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:51.211024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c93d88-db41-49ad-9c97-842f304e81ea None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:51.211512 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c93d88-db41-49ad-9c97-842f304e81ea None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:51.211984 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c93d88-db41-49ad-9c97-842f304e81ea None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:51.238939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1668e69d-8472-4abd-8b88-5ecd9da8a5c4 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:51.240528 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c93d88-db41-49ad-9c97-842f304e81ea None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:51.242012 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c93d88-db41-49ad-9c97-842f304e81ea None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:51.272652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-98c93d88-db41-49ad-9c97-842f304e81ea None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:51.272652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:51.272652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:51.272652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:51.272652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:51.272652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:51.272652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:51.272652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:51.272652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:51.272652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:51.272652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:51.272652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:51.272652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:51.272652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:51.272652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:51.272652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:51.274101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:51.274101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:51.274101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:51.274101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:51.274101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:51.274101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:51.274101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:51.274101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:51.274101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:51.274101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:51.274101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:51.274101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:51.274101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:51.274101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:51.274101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:51.274101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:51.274101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:51.275042 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:51.275042 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:51.275042 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:51.275042 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 345/684] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:51.279616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c93d88-db41-49ad-9c97-842f304e81ea None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:51.307716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fa223b-3603-49a6-8df1-6ea3d8e295c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:51.309076 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fa223b-3603-49a6-8df1-6ea3d8e295c5 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:51.309363 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fa223b-3603-49a6-8df1-6ea3d8e295c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:51.309640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fa223b-3603-49a6-8df1-6ea3d8e295c5 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:51.310315 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fa223b-3603-49a6-8df1-6ea3d8e295c5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:51.310621 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fa223b-3603-49a6-8df1-6ea3d8e295c5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:51.312290 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fa223b-3603-49a6-8df1-6ea3d8e295c5 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:51.324917 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 340/685] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:51.337518 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fa223b-3603-49a6-8df1-6ea3d8e295c5 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:51.432874 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c93d88-db41-49ad-9c97-842f304e81ea None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:51.459463 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fa223b-3603-49a6-8df1-6ea3d8e295c5 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:51.472297 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0975c96e-0aea-465d-bd7c-79a2050c5ce2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:51.473220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0975c96e-0aea-465d-bd7c-79a2050c5ce2 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:51.473705 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0975c96e-0aea-465d-bd7c-79a2050c5ce2 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:51.474230 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0975c96e-0aea-465d-bd7c-79a2050c5ce2 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:51.475206 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0975c96e-0aea-465d-bd7c-79a2050c5ce2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:51.475738 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0975c96e-0aea-465d-bd7c-79a2050c5ce2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:51.477441 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0975c96e-0aea-465d-bd7c-79a2050c5ce2 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:51.491540 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 346/686] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:51.497009 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0975c96e-0aea-465d-bd7c-79a2050c5ce2 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:51.498195 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c63f3ca-d3a9-4799-a851-5d5ca3e0b465 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:51.498839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c63f3ca-d3a9-4799-a851-5d5ca3e0b465 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:51.499175 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c63f3ca-d3a9-4799-a851-5d5ca3e0b465 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:51.499454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c63f3ca-d3a9-4799-a851-5d5ca3e0b465 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:51.508212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:51.512910 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c63f3ca-d3a9-4799-a851-5d5ca3e0b465 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:51.520027 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c63f3ca-d3a9-4799-a851-5d5ca3e0b465 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:51.520150 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:51.531423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-4c63f3ca-d3a9-4799-a851-5d5ca3e0b465 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:51.531423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:51.531423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:51.531423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:51.531423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:51.531423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:51.531423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:51.531423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:51.531423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:51.531423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:51.531423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:51.531423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:51.531423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:51.531423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:51.531423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:51.531423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:51.532924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:51.532924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:51.532924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:51.532924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:51.532924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:51.532924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:51.532924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:51.532924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:51.532924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:51.532924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:51.532924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:51.532924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:51.532924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:51.532924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:51.532924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:51.532924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:51.532924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:51.533822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:51.533822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:51.533822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:51.533822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 341/687] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:51.538819 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c63f3ca-d3a9-4799-a851-5d5ca3e0b465 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:51.611717 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0975c96e-0aea-465d-bd7c-79a2050c5ce2 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:51.647628 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3cee06-4b4e-46d0-9f2f-871270de18f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:51.648623 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3cee06-4b4e-46d0-9f2f-871270de18f6 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:51.649109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3cee06-4b4e-46d0-9f2f-871270de18f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:51.649607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3cee06-4b4e-46d0-9f2f-871270de18f6 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:51.657527 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c63f3ca-d3a9-4799-a851-5d5ca3e0b465 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:51.658619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3cee06-4b4e-46d0-9f2f-871270de18f6 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:51.660302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3cee06-4b4e-46d0-9f2f-871270de18f6 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:51.672799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-5f3cee06-4b4e-46d0-9f2f-871270de18f6 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 16 03:27:51.672799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:51.672799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:51.672799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:51.672799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:51.672799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:51.672799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:51.672799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:51.672799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:51.672799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:51.672799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:51.672799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:51.672799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:27:51.672799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:27:51.672799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:27:51.672799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:27:51.673810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:51.673810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:51.673810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:27:51.673810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:51.673810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:27:51.673810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:51.673810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:51.673810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:51.673810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:51.673810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:51.673810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:51.673810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:51.673810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:51.673810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:51.673810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:51.673810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:51.673810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:51.673810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:51.674780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:51.674780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:51.674780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:27:51.674780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:27:51.674780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:27:51.674780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:27:51.674780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:27:51.674780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:27:51.674780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 16 03:27:51.674780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:51.677304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 347/688] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:27:51 2022] GET /identity/v3/users/swift => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:51.682730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3cee06-4b4e-46d0-9f2f-871270de18f6 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:51.695831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d88709-f282-4c26-9982-a5b7bb4d2ce2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:51.696459 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d88709-f282-4c26-9982-a5b7bb4d2ce2 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:51.696795 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d88709-f282-4c26-9982-a5b7bb4d2ce2 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:51.697079 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d88709-f282-4c26-9982-a5b7bb4d2ce2 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:51.697747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d88709-f282-4c26-9982-a5b7bb4d2ce2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:51.698104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d88709-f282-4c26-9982-a5b7bb4d2ce2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:51.699643 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d88709-f282-4c26-9982-a5b7bb4d2ce2 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:51.716339 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 342/689] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:51.722248 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d88709-f282-4c26-9982-a5b7bb4d2ce2 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:51.773756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:51.790090 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:51.800688 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3cee06-4b4e-46d0-9f2f-871270de18f6 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:51.840619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c7a56d-c609-4a4e-91ec-851064ad8597 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:51.842362 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c7a56d-c609-4a4e-91ec-851064ad8597 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:51.843529 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c7a56d-c609-4a4e-91ec-851064ad8597 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:51.844218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c7a56d-c609-4a4e-91ec-851064ad8597 None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:51.846109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c7a56d-c609-4a4e-91ec-851064ad8597 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:51.846808 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c7a56d-c609-4a4e-91ec-851064ad8597 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:51.849212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c7a56d-c609-4a4e-91ec-851064ad8597 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:51.857041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d88709-f282-4c26-9982-a5b7bb4d2ce2 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:51.885983 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 348/690] 10.222.0.49 () {56 vars in 1135 bytes} [Fri Sep 16 03:27:51 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:51.890217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c7a56d-c609-4a4e-91ec-851064ad8597 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:51.934716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0502d4c-b087-4a69-99ff-6d2e01197031 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:51.935763 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0502d4c-b087-4a69-99ff-6d2e01197031 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:51.936245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0502d4c-b087-4a69-99ff-6d2e01197031 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:51.936732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0502d4c-b087-4a69-99ff-6d2e01197031 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:51.948393 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0502d4c-b087-4a69-99ff-6d2e01197031 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:51.955677 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0502d4c-b087-4a69-99ff-6d2e01197031 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:51.977147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-f0502d4c-b087-4a69-99ff-6d2e01197031 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 16 03:27:51.977147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:51.977147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:51.977147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:51.977147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:51.977147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:51.977147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:51.977147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:51.977147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:51.977147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:51.977147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:51.977147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:51.977147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:27:51.977147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:27:51.977147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:27:51.977147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:27:51.980753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:51.980753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:51.980753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:27:51.980753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:51.980753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:27:51.980753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:51.980753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:51.980753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:51.980753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:51.980753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:51.980753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:51.980753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:51.980753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:51.980753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:51.980753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:51.980753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:51.980753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:51.980753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:51.981797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:51.981797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:51.981797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:27:51.981797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:27:51.981797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:27:51.981797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:27:51.981797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:27:51.981797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:27:51.981797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 16 03:27:51.981797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:51.992513 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 343/691] 10.222.0.49 () {56 vars in 1093 bytes} [Fri Sep 16 03:27:51 2022] GET /identity/v3/users/nova => generated 82 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:51.999086 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0502d4c-b087-4a69-99ff-6d2e01197031 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:52.063371 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c7a56d-c609-4a4e-91ec-851064ad8597 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:52.122720 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffc808d-8e30-4d4b-8976-5e0df1c6d1ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:52.123397 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffc808d-8e30-4d4b-8976-5e0df1c6d1ae None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:52.123682 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffc808d-8e30-4d4b-8976-5e0df1c6d1ae None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:52.123958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffc808d-8e30-4d4b-8976-5e0df1c6d1ae None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:52.127002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:52.137530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0502d4c-b087-4a69-99ff-6d2e01197031 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:52.148697 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:52.173902 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c22891-0a51-4bf2-93e3-a04b92da4f7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:52.174515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c22891-0a51-4bf2-93e3-a04b92da4f7c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:52.174792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c22891-0a51-4bf2-93e3-a04b92da4f7c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:52.175095 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c22891-0a51-4bf2-93e3-a04b92da4f7c None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:52.175932 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c22891-0a51-4bf2-93e3-a04b92da4f7c None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:52.176236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c22891-0a51-4bf2-93e3-a04b92da4f7c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:52.177771 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c22891-0a51-4bf2-93e3-a04b92da4f7c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:52.197165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 344/692] 10.222.0.49 () {56 vars in 1133 bytes} [Fri Sep 16 03:27:51 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:52.206454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c22891-0a51-4bf2-93e3-a04b92da4f7c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:52.243475 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffc808d-8e30-4d4b-8976-5e0df1c6d1ae None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:52.247281 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffc808d-8e30-4d4b-8976-5e0df1c6d1ae None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:52.396023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c22891-0a51-4bf2-93e3-a04b92da4f7c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:52.435499 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 349/693] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 545 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 16 03:27:52.442192 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffc808d-8e30-4d4b-8976-5e0df1c6d1ae None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:52.466288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a49bae1-ebd6-4f2e-90f0-8743ff5c7dd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:52.466925 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a49bae1-ebd6-4f2e-90f0-8743ff5c7dd7 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:52.467234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a49bae1-ebd6-4f2e-90f0-8743ff5c7dd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:52.467506 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a49bae1-ebd6-4f2e-90f0-8743ff5c7dd7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:52.515280 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:52.550826 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:52.580416 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffc808d-8e30-4d4b-8976-5e0df1c6d1ae None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:52.630365 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a49bae1-ebd6-4f2e-90f0-8743ff5c7dd7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:52.631970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a49bae1-ebd6-4f2e-90f0-8743ff5c7dd7 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:52.633049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0a2c6d-39e5-43c8-a745-f7790924439e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:52.634026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0a2c6d-39e5-43c8-a745-f7790924439e None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:52.634536 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0a2c6d-39e5-43c8-a745-f7790924439e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:52.635100 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0a2c6d-39e5-43c8-a745-f7790924439e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:52.646538 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0a2c6d-39e5-43c8-a745-f7790924439e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:52.648444 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0a2c6d-39e5-43c8-a745-f7790924439e None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:52.664116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-bd0a2c6d-39e5-43c8-a745-f7790924439e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:52.664116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:52.664116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:52.664116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:52.664116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:52.664116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:52.664116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:52.664116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:52.664116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:52.664116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:52.664116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:52.664116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:52.664116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:52.664116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:52.664116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:52.664116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:52.665141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:52.665141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:52.665141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:52.665141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:52.665141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:52.665141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:52.665141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:52.665141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:52.665141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:52.665141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:52.665141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:52.665141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:52.665141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:52.665141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:52.665141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:52.665141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:52.665141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:52.666061 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:52.666061 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:52.666061 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:52.668214 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 350/694] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:27:52.676743 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0a2c6d-39e5-43c8-a745-f7790924439e None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:52.776933 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 345/695] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 571 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 16 03:27:52.783195 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a49bae1-ebd6-4f2e-90f0-8743ff5c7dd7 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:52.855309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:52.858644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:52.873680 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0a2c6d-39e5-43c8-a745-f7790924439e None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:52.899940 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a49bae1-ebd6-4f2e-90f0-8743ff5c7dd7 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:52.912333 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b83784-41ca-40e7-acff-aa98028983ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:52.913039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b83784-41ca-40e7-acff-aa98028983ea None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:52.913390 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b83784-41ca-40e7-acff-aa98028983ea None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:52.913748 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b83784-41ca-40e7-acff-aa98028983ea None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:52.914450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b83784-41ca-40e7-acff-aa98028983ea None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:52.914937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b83784-41ca-40e7-acff-aa98028983ea None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:52.916314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b83784-41ca-40e7-acff-aa98028983ea None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:52.929586 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 351/696] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:52.931126 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7d7193-ddfe-489b-b64f-26664e2daf4c None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:52.931476 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7d7193-ddfe-489b-b64f-26664e2daf4c None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:52.931825 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7d7193-ddfe-489b-b64f-26664e2daf4c None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:52.937217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 352/697] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:52.937217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7d7193-ddfe-489b-b64f-26664e2daf4c None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:52.947680 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052b1266-9266-4629-9bcb-0e031c8d0fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:52.948850 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052b1266-9266-4629-9bcb-0e031c8d0fb9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:52.950459 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052b1266-9266-4629-9bcb-0e031c8d0fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:52.950459 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052b1266-9266-4629-9bcb-0e031c8d0fb9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:52.963040 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052b1266-9266-4629-9bcb-0e031c8d0fb9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:52.965476 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052b1266-9266-4629-9bcb-0e031c8d0fb9 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:52.986505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-052b1266-9266-4629-9bcb-0e031c8d0fb9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:52.986505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:52.986505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:52.986505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:52.986505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:52.986505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:52.986505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:52.986505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:52.986505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:52.986505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:52.986505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:52.986505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:52.986505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:52.986505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:52.986505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:52.986505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:52.987839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:52.987839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:52.987839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:52.987839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:52.987839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:52.987839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:52.987839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:52.987839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:52.987839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:52.987839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:52.987839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:52.987839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:52.987839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:52.987839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:52.987839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:52.987839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:52.987839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:52.988731 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:52.988731 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:52.988731 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:52.988731 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 346/698] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:52.989523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1426dfd0-55d2-4405-9543-3b0813634e31 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:52.989523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1426dfd0-55d2-4405-9543-3b0813634e31 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:52.989693 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1426dfd0-55d2-4405-9543-3b0813634e31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:53.012003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1426dfd0-55d2-4405-9543-3b0813634e31 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:53.013456 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1426dfd0-55d2-4405-9543-3b0813634e31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:53.058773 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7d7193-ddfe-489b-b64f-26664e2daf4c None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:53.097228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b18c08-83fa-47d0-a6ad-7639d8b47930 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:53.098299 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b18c08-83fa-47d0-a6ad-7639d8b47930 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:53.098778 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b18c08-83fa-47d0-a6ad-7639d8b47930 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:53.099301 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b18c08-83fa-47d0-a6ad-7639d8b47930 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:53.107207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b18c08-83fa-47d0-a6ad-7639d8b47930 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:53.109009 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b18c08-83fa-47d0-a6ad-7639d8b47930 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:53.111268 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 347/699] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:53.112399 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1426dfd0-55d2-4405-9543-3b0813634e31 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:53.123320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-27b18c08-83fa-47d0-a6ad-7639d8b47930 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:53.123320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:53.123320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:53.123320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:53.123320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:53.123320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:53.123320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:53.123320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:53.123320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:53.123320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:53.123320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:53.123320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:53.123320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:27:53.123320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:27:53.123320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:27:53.124223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:27:53.124223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:53.124223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:53.124223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:53.124223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:53.124223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:53.124223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:53.124223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:53.124223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:53.124223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:53.124223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:53.124223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:53.124223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:53.124223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:53.124223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:53.124223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:53.124223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:53.125125 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:27:53.125125 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:27:53.125125 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:27:53.125125 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:27:53.125125 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:27:53.125125 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:27:53.125125 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:53.125125 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:53.127980 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 353/700] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:53.135914 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b18c08-83fa-47d0-a6ad-7639d8b47930 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:53.151178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:53.179301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:53.232036 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1426dfd0-55d2-4405-9543-3b0813634e31 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:53.282229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f26fcb-b6fc-4e15-ab4c-bd69afa0bcd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:53.282853 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f26fcb-b6fc-4e15-ab4c-bd69afa0bcd3 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:53.283155 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f26fcb-b6fc-4e15-ab4c-bd69afa0bcd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:53.283429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f26fcb-b6fc-4e15-ab4c-bd69afa0bcd3 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:53.284625 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f26fcb-b6fc-4e15-ab4c-bd69afa0bcd3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:53.284931 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f26fcb-b6fc-4e15-ab4c-bd69afa0bcd3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:53.286827 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f26fcb-b6fc-4e15-ab4c-bd69afa0bcd3 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:53.296355 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b18c08-83fa-47d0-a6ad-7639d8b47930 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:53.303033 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 348/701] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:53.304352 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67c0808-cfa6-4079-a49e-88c82c096079 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:53.304644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67c0808-cfa6-4079-a49e-88c82c096079 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:53.304919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67c0808-cfa6-4079-a49e-88c82c096079 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:53.306346 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 349/702] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:53.311186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67c0808-cfa6-4079-a49e-88c82c096079 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:53.343981 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d6860a-9729-41dc-b151-3d0d5c1aa556 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:53.345123 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d6860a-9729-41dc-b151-3d0d5c1aa556 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:53.345498 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d6860a-9729-41dc-b151-3d0d5c1aa556 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:53.345789 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d6860a-9729-41dc-b151-3d0d5c1aa556 None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:53.346579 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d6860a-9729-41dc-b151-3d0d5c1aa556 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:53.346893 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d6860a-9729-41dc-b151-3d0d5c1aa556 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:53.348585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d6860a-9729-41dc-b151-3d0d5c1aa556 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:53.362436 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 354/703] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:27:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:53.363914 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7771ea-33bf-4bc1-9bf9-60e67057977d None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:53.364206 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7771ea-33bf-4bc1-9bf9-60e67057977d None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:53.364486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7771ea-33bf-4bc1-9bf9-60e67057977d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:53.384433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-ce7771ea-33bf-4bc1-9bf9-60e67057977d None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:53.385686 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ce7771ea-33bf-4bc1-9bf9-60e67057977d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:53.443576 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67c0808-cfa6-4079-a49e-88c82c096079 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:53.452607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:53.456157 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:53.464904 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 355/704] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:53.464904 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7771ea-33bf-4bc1-9bf9-60e67057977d None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:53.485114 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3ee5a3-98e9-43ee-969a-23280831a06e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:53.486047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3ee5a3-98e9-43ee-969a-23280831a06e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:53.486332 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3ee5a3-98e9-43ee-969a-23280831a06e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:53.486607 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3ee5a3-98e9-43ee-969a-23280831a06e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:53.494066 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3ee5a3-98e9-43ee-969a-23280831a06e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:53.495967 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3ee5a3-98e9-43ee-969a-23280831a06e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:53.509356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-bd3ee5a3-98e9-43ee-969a-23280831a06e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:53.509356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:53.509356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:53.509356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:53.509356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:53.509356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:53.509356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:53.509356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:53.509356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:53.509356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:53.509356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:53.509356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:53.509356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:27:53.509356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:27:53.509356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:27:53.510299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:27:53.510299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:53.510299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:53.510299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:53.510299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:53.510299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:53.510299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:53.510299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:53.510299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:53.510299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:53.510299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:53.510299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:53.510299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:53.510299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:53.510299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:53.510299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:53.510299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:53.511251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:27:53.511251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:27:53.511251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:27:53.511251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:27:53.511251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:27:53.511251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:27:53.511251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:27:53.511251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:53.511251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 350/705] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:53.511743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3ee5a3-98e9-43ee-969a-23280831a06e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:53.593644 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7771ea-33bf-4bc1-9bf9-60e67057977d None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:53.633852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3ee5a3-98e9-43ee-969a-23280831a06e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:53.639110 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c46c9c0-f4a4-4d3d-9bdb-db17726acb1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:53.639897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c46c9c0-f4a4-4d3d-9bdb-db17726acb1e None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:53.640373 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c46c9c0-f4a4-4d3d-9bdb-db17726acb1e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:53.641863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c46c9c0-f4a4-4d3d-9bdb-db17726acb1e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:53.642861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c46c9c0-f4a4-4d3d-9bdb-db17726acb1e None admin] RBAC: Adding query filter params (role.id=974d839067e9411aa4cb850cee33f6ae, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=aa30da9972e140d7905c522576681429) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:53.643825 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c46c9c0-f4a4-4d3d-9bdb-db17726acb1e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:53.646121 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c46c9c0-f4a4-4d3d-9bdb-db17726acb1e None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:53.654437 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 356/706] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:27:53 2022] GET /identity/v3/role_assignments?user.id=aa30da9972e140d7905c522576681429&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=974d839067e9411aa4cb850cee33f6ae => generated 608 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:53.656283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c46c9c0-f4a4-4d3d-9bdb-db17726acb1e None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:53.671031 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7adf4da-9b00-4a5a-aa35-61069b90fb23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:53.671816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7adf4da-9b00-4a5a-aa35-61069b90fb23 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:53.672629 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7adf4da-9b00-4a5a-aa35-61069b90fb23 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:53.672904 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7adf4da-9b00-4a5a-aa35-61069b90fb23 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:53.694263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7adf4da-9b00-4a5a-aa35-61069b90fb23 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:53.695830 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7adf4da-9b00-4a5a-aa35-61069b90fb23 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:53.721324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-e7adf4da-9b00-4a5a-aa35-61069b90fb23 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:53.721324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:53.721324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:53.721324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:53.721324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:53.721324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:53.721324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:53.721324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:53.721324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:53.721324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:53.721324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:53.721324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:53.721324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:53.721324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:53.721324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:53.721324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:53.722425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:53.722425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:53.722425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:53.722425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:53.722425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:53.722425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:53.722425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:53.722425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:53.722425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:53.722425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:53.722425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:53.722425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:53.722425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:53.722425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:53.722425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:53.722425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:53.722425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:53.723388 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:53.723388 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:53.723388 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:53.723388 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 351/707] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:53.732033 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7adf4da-9b00-4a5a-aa35-61069b90fb23 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:53.778076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:53.787279 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:53.831508 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c46c9c0-f4a4-4d3d-9bdb-db17726acb1e None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:53.871574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158e3022-54bd-47db-95c4-6fb21c5f7630 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:53.872255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158e3022-54bd-47db-95c4-6fb21c5f7630 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:53.872536 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158e3022-54bd-47db-95c4-6fb21c5f7630 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:53.872809 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158e3022-54bd-47db-95c4-6fb21c5f7630 None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:53.873558 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158e3022-54bd-47db-95c4-6fb21c5f7630 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:53.873876 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158e3022-54bd-47db-95c4-6fb21c5f7630 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:53.875400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158e3022-54bd-47db-95c4-6fb21c5f7630 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:53.894738 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7adf4da-9b00-4a5a-aa35-61069b90fb23 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:53.895277 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 357/708] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:27:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:53.896200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158e3022-54bd-47db-95c4-6fb21c5f7630 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:53.937083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ed76c5-81b6-44a9-b1c4-f738b6094751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:53.938243 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ed76c5-81b6-44a9-b1c4-f738b6094751 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:53.938573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ed76c5-81b6-44a9-b1c4-f738b6094751 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:53.939097 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ed76c5-81b6-44a9-b1c4-f738b6094751 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:53.939897 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ed76c5-81b6-44a9-b1c4-f738b6094751 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:53.940208 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ed76c5-81b6-44a9-b1c4-f738b6094751 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:53.942037 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ed76c5-81b6-44a9-b1c4-f738b6094751 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:53.953587 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 352/709] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:53.954680 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ed76c5-81b6-44a9-b1c4-f738b6094751 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:54.007556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158e3022-54bd-47db-95c4-6fb21c5f7630 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:54.032854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:54.046724 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee095388-ae41-4f38-be69-5df2fe2ecb87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:54.047835 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee095388-ae41-4f38-be69-5df2fe2ecb87 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:54.048719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee095388-ae41-4f38-be69-5df2fe2ecb87 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:54.048719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee095388-ae41-4f38-be69-5df2fe2ecb87 None admin] PATH_INFO: `/v3/services/network` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:54.049249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee095388-ae41-4f38-be69-5df2fe2ecb87 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:54.050648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee095388-ae41-4f38-be69-5df2fe2ecb87 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:54.051313 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:54.061345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-ee095388-ae41-4f38-be69-5df2fe2ecb87 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 16 03:27:54.061345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:54.061345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:27:54.061345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:27:54.061345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:27:54.061345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:27:54.061345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:27:54.061345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:27:54.061345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 16 03:27:54.061345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:54.061345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:27:54.061345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:54.061345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:54.061345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:54.061345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:54.061345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:54.061345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:54.062381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:54.062381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:54.062381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:54.062381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:54.062381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:54.062381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:54.062381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:27:54.062381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:27:54.062381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:27:54.062381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:27:54.062381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:54.062381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:54.062381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:54.062381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:54.062381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:54.062381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:54.063323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:54.063323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:54.063323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:54.063323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:54.063323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:54.063323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:54.063323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:54.063323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:54.063323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:54.063323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:54.063323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:27:54.063323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:27:54.063323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 16 03:27:54.063323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:54.063323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 358/710] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:53 2022] GET /identity/v3/services/network => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:54.064681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f583a6-1a6f-4e23-b8b8-23ce46436168 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:54.064681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f583a6-1a6f-4e23-b8b8-23ce46436168 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:54.064856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f583a6-1a6f-4e23-b8b8-23ce46436168 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:54.070173 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 359/711] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:54 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:54.070927 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f583a6-1a6f-4e23-b8b8-23ce46436168 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:54.075170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ed76c5-81b6-44a9-b1c4-f738b6094751 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:54.112202 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddeaf6f0-76d3-4d41-a10e-ab92cf4f7a4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:54.112820 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeaf6f0-76d3-4d41-a10e-ab92cf4f7a4d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:54.113103 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeaf6f0-76d3-4d41-a10e-ab92cf4f7a4d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:54.113382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeaf6f0-76d3-4d41-a10e-ab92cf4f7a4d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:54.114111 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddeaf6f0-76d3-4d41-a10e-ab92cf4f7a4d None admin] RBAC: Adding query filter params (role.id=89c4b3a971224d468d33d2dc411230a2, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=ded1bcd10e62499a9aaa0ece5aa55c80) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:54.114422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddeaf6f0-76d3-4d41-a10e-ab92cf4f7a4d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:54.115887 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddeaf6f0-76d3-4d41-a10e-ab92cf4f7a4d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:54.121592 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 353/712] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:27:53 2022] GET /identity/v3/role_assignments?user.id=ded1bcd10e62499a9aaa0ece5aa55c80&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=89c4b3a971224d468d33d2dc411230a2 => generated 259 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:54.122320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddeaf6f0-76d3-4d41-a10e-ab92cf4f7a4d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:54.224028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f583a6-1a6f-4e23-b8b8-23ce46436168 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:54.244989 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddeaf6f0-76d3-4d41-a10e-ab92cf4f7a4d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:54.265355 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e59a4fe-1f42-41f2-a94d-7e9bec21a6e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:54.266731 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e59a4fe-1f42-41f2-a94d-7e9bec21a6e2 None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:54.267064 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e59a4fe-1f42-41f2-a94d-7e9bec21a6e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:54.267345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e59a4fe-1f42-41f2-a94d-7e9bec21a6e2 None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:54.268172 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e59a4fe-1f42-41f2-a94d-7e9bec21a6e2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:54.269745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e59a4fe-1f42-41f2-a94d-7e9bec21a6e2 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:54.293488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46441797-7a3e-4faa-adc7-e94acffcf930 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:54.294117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46441797-7a3e-4faa-adc7-e94acffcf930 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:54.294398 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46441797-7a3e-4faa-adc7-e94acffcf930 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:54.294671 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46441797-7a3e-4faa-adc7-e94acffcf930 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:54.295345 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46441797-7a3e-4faa-adc7-e94acffcf930 None admin] RBAC: Adding query filter params (name=network) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:54.295643 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46441797-7a3e-4faa-adc7-e94acffcf930 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:54.307280 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46441797-7a3e-4faa-adc7-e94acffcf930 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:54.307280 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 354/713] 10.222.0.49 () {56 vars in 1112 bytes} [Fri Sep 16 03:27:54 2022] GET /identity/v3/services?name=network => generated 124 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:54.307280 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfcee89-d1b9-49bb-ae0c-54eed3364757 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:54.307280 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfcee89-d1b9-49bb-ae0c-54eed3364757 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:54.307280 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfcee89-d1b9-49bb-ae0c-54eed3364757 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:54.325706 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1bfcee89-d1b9-49bb-ae0c-54eed3364757 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:54.327391 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1bfcee89-d1b9-49bb-ae0c-54eed3364757 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:54.331724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:54.399931 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 355/715] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:54.401060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfcee89-d1b9-49bb-ae0c-54eed3364757 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:54.406893 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 360/715] 10.222.0.49 () {60 vars in 1132 bytes} [Fri Sep 16 03:27:54 2022] POST /identity/v3/users => generated 254 bytes in 329 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:27:54.515918 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfcee89-d1b9-49bb-ae0c-54eed3364757 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:54.554718 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215305ee-ffd1-4ab3-a329-5613ea98ab51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:54.555598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215305ee-ffd1-4ab3-a329-5613ea98ab51 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:54.556020 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215305ee-ffd1-4ab3-a329-5613ea98ab51 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:54.556461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215305ee-ffd1-4ab3-a329-5613ea98ab51 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:54.557235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215305ee-ffd1-4ab3-a329-5613ea98ab51 None admin] RBAC: Adding query filter params (type=network) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:54.557674 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215305ee-ffd1-4ab3-a329-5613ea98ab51 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:54.559192 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215305ee-ffd1-4ab3-a329-5613ea98ab51 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:54.566440 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 356/716] 10.222.0.49 () {56 vars in 1112 bytes} [Fri Sep 16 03:27:54 2022] GET /identity/v3/services?type=network => generated 124 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:54.769868 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e59a4fe-1f42-41f2-a94d-7e9bec21a6e2 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:54.818697 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:54.879362 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e59a4fe-1f42-41f2-a94d-7e9bec21a6e2 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:54.916145 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7339327b-7156-4d70-a517-7b73e0d267a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:54.916780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7339327b-7156-4d70-a517-7b73e0d267a4 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:54.917058 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7339327b-7156-4d70-a517-7b73e0d267a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:54.917327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7339327b-7156-4d70-a517-7b73e0d267a4 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:54.917900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7339327b-7156-4d70-a517-7b73e0d267a4 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:54.919310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7339327b-7156-4d70-a517-7b73e0d267a4 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:54.928949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-7339327b-7156-4d70-a517-7b73e0d267a4 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 16 03:27:54.928949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:54.928949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:27:54.928949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:27:54.928949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:27:54.928949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:27:54.928949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:27:54.928949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:27:54.928949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 16 03:27:54.928949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:54.928949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:27:54.928949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:54.928949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:54.928949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:54.928949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:54.928949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:54.928949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:54.929948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:54.929948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:54.929948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:54.929948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:54.929948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:54.929948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:54.929948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:27:54.929948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:27:54.929948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:27:54.929948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:27:54.929948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:54.929948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:54.929948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:54.929948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:54.929948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:54.929948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:54.930790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:54.930790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:54.930790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:54.930790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:54.930790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:54.930790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:54.930790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:54.930790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:54.930790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:54.930790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:54.930790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:27:54.930790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:27:54.930790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 16 03:27:54.930790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:54.931552 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 361/717] 10.222.0.49 () {56 vars in 1129 bytes} [Fri Sep 16 03:27:54 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:54.936675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215305ee-ffd1-4ab3-a329-5613ea98ab51 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:54.941138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:55.046775 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215305ee-ffd1-4ab3-a329-5613ea98ab51 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:55.089852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4206661d-0295-41f7-8179-bc87a864bf36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:55.090832 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4206661d-0295-41f7-8179-bc87a864bf36 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:55.091364 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4206661d-0295-41f7-8179-bc87a864bf36 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:55.091861 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4206661d-0295-41f7-8179-bc87a864bf36 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:55.092679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4206661d-0295-41f7-8179-bc87a864bf36 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:55.093231 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4206661d-0295-41f7-8179-bc87a864bf36 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:55.094875 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4206661d-0295-41f7-8179-bc87a864bf36 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:55.099613 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 357/718] 10.222.0.49 () {56 vars in 1124 bytes} [Fri Sep 16 03:27:54 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:55.105337 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7339327b-7156-4d70-a517-7b73e0d267a4 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:55.258132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7339327b-7156-4d70-a517-7b73e0d267a4 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:55.260474 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:55.301302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023b1e13-99f5-438e-8b27-f57a50ebc3ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:55.301827 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023b1e13-99f5-438e-8b27-f57a50ebc3ee None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:55.302104 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023b1e13-99f5-438e-8b27-f57a50ebc3ee None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:55.302375 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023b1e13-99f5-438e-8b27-f57a50ebc3ee None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:55.303014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023b1e13-99f5-438e-8b27-f57a50ebc3ee None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:55.303318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023b1e13-99f5-438e-8b27-f57a50ebc3ee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:55.304678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023b1e13-99f5-438e-8b27-f57a50ebc3ee None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:55.316826 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 362/719] 10.222.0.49 () {56 vars in 1124 bytes} [Fri Sep 16 03:27:55 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:56.275977 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d3eab2-3f92-427a-9024-a3b08ad801d8 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:56.276510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d3eab2-3f92-427a-9024-a3b08ad801d8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:56.276510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d3eab2-3f92-427a-9024-a3b08ad801d8 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:56.278147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 358/720] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:56.285076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe93fd3-0273-4090-8141-01be2bfcb246 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:56.285483 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe93fd3-0273-4090-8141-01be2bfcb246 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:56.285650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe93fd3-0273-4090-8141-01be2bfcb246 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:56.305956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-0fe93fd3-0273-4090-8141-01be2bfcb246 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:56.309437 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-0fe93fd3-0273-4090-8141-01be2bfcb246 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:56.415220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 363/721] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 03:27:56.613014 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a9f9b7-abf2-4d2d-adb1-72be88146f14 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:56.613355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a9f9b7-abf2-4d2d-adb1-72be88146f14 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:56.613470 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a9f9b7-abf2-4d2d-adb1-72be88146f14 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:56.614794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 359/722] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:56.620152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8017c0ea-0dfe-4893-90ca-cb0f2dd0a665 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:56.620444 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8017c0ea-0dfe-4893-90ca-cb0f2dd0a665 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:56.620720 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8017c0ea-0dfe-4893-90ca-cb0f2dd0a665 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:56.647945 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-8017c0ea-0dfe-4893-90ca-cb0f2dd0a665 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:56.649138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-8017c0ea-0dfe-4893-90ca-cb0f2dd0a665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:56.736141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 364/723] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:56.879820 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a9f9b7-abf2-4d2d-adb1-72be88146f14 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:56.965256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:57.009596 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a9f9b7-abf2-4d2d-adb1-72be88146f14 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:57.055547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbc36ab-fc83-45a8-8b02-26cb2e34b38b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:57.056182 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbc36ab-fc83-45a8-8b02-26cb2e34b38b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:57.056461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbc36ab-fc83-45a8-8b02-26cb2e34b38b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:57.056732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbc36ab-fc83-45a8-8b02-26cb2e34b38b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:57.071804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbc36ab-fc83-45a8-8b02-26cb2e34b38b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:57.077252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-cfbc36ab-fc83-45a8-8b02-26cb2e34b38b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:27:57.077252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:57.077252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:57.077252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:57.077252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:57.077252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:57.077252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:57.077252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:57.077252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:57.077252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:57.077252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:57.077252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:57.077252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:57.077252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:57.077252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:57.077252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:57.078186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:57.078186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:57.078186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:57.078186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:57.078186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:57.078186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:57.078186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:57.078186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:57.078186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:57.078186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:57.078186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:57.078186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:57.078186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:57.078186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:57.078186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:57.078186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:57.079684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:57.079684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:57.079684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:57.079684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:57.079684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:57.079684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:57.079684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:57.079684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:57.079684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:57.079684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:57.079684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:27:57.079684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:57.079684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 360/724] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:27:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:57.087515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8017c0ea-0dfe-4893-90ca-cb0f2dd0a665 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:57.096376 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbc36ab-fc83-45a8-8b02-26cb2e34b38b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:57.145242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:57.208198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbc36ab-fc83-45a8-8b02-26cb2e34b38b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:57.244201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271d8e3b-774b-4459-9c92-e259d3042e5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:57.245193 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271d8e3b-774b-4459-9c92-e259d3042e5d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:57.245670 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271d8e3b-774b-4459-9c92-e259d3042e5d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:57.246135 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271d8e3b-774b-4459-9c92-e259d3042e5d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:57.247278 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:57.258854 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271d8e3b-774b-4459-9c92-e259d3042e5d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:57.264023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-271d8e3b-774b-4459-9c92-e259d3042e5d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:27:57.264023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:57.264023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:57.264023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:57.264023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:57.264023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:57.264023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:57.264023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:57.264023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:57.264023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:57.264023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:57.264023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:57.264023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:57.264023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:57.264023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:57.264023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:57.264944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:57.264944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:57.264944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:57.264944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:57.264944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:57.264944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:57.264944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:57.264944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:57.264944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:57.264944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:57.264944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:57.264944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:57.264944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:57.264944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:57.264944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:57.264944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:57.266560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:57.266560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:57.266560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:57.266560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:57.266560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:57.266560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:57.266560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:57.266560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:57.266560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:57.266560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:57.266560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:27:57.266560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:57.268994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 361/725] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:27:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:57.274184 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271d8e3b-774b-4459-9c92-e259d3042e5d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:57.393423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8017c0ea-0dfe-4893-90ca-cb0f2dd0a665 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:57.399359 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271d8e3b-774b-4459-9c92-e259d3042e5d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:57.458189 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af61865f-e459-42c1-8cfd-6b8a56ac4af1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:57.459206 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af61865f-e459-42c1-8cfd-6b8a56ac4af1 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:57.459713 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af61865f-e459-42c1-8cfd-6b8a56ac4af1 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:57.460357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af61865f-e459-42c1-8cfd-6b8a56ac4af1 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:57.461584 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af61865f-e459-42c1-8cfd-6b8a56ac4af1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:57.462111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af61865f-e459-42c1-8cfd-6b8a56ac4af1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:57.463739 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af61865f-e459-42c1-8cfd-6b8a56ac4af1 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:57.474429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 365/726] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 387 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:57.479102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade03dac-cada-4268-bd1c-71b2a4ceaef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:57.480201 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af61865f-e459-42c1-8cfd-6b8a56ac4af1 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:57.481749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade03dac-cada-4268-bd1c-71b2a4ceaef9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:57.484045 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade03dac-cada-4268-bd1c-71b2a4ceaef9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:57.485263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade03dac-cada-4268-bd1c-71b2a4ceaef9 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:57.486365 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade03dac-cada-4268-bd1c-71b2a4ceaef9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:57.491397 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade03dac-cada-4268-bd1c-71b2a4ceaef9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:57.493005 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade03dac-cada-4268-bd1c-71b2a4ceaef9 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:57.507029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 362/727] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:27:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:27:57.513214 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade03dac-cada-4268-bd1c-71b2a4ceaef9 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:57.575501 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:57.603035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:57.613863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af61865f-e459-42c1-8cfd-6b8a56ac4af1 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:57.637768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade03dac-cada-4268-bd1c-71b2a4ceaef9 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:57.651016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec60b645-894f-4b22-b780-db27d0e6086c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:57.651596 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec60b645-894f-4b22-b780-db27d0e6086c None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:57.651877 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec60b645-894f-4b22-b780-db27d0e6086c None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:57.652152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec60b645-894f-4b22-b780-db27d0e6086c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:57.664468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec60b645-894f-4b22-b780-db27d0e6086c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:57.665909 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec60b645-894f-4b22-b780-db27d0e6086c None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:57.678743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff5f54d-76bd-4170-a5c7-984af67b66ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:57.679444 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff5f54d-76bd-4170-a5c7-984af67b66ff None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:57.679718 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff5f54d-76bd-4170-a5c7-984af67b66ff None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:57.679984 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff5f54d-76bd-4170-a5c7-984af67b66ff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:57.681643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-ec60b645-894f-4b22-b780-db27d0e6086c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:57.681643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:57.681643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:57.681643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:57.681643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:57.681643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:57.681643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:57.681643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:57.681643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:57.681643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:57.681643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:57.681643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:57.681643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:57.681643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:57.681643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:57.681643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:57.682562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:57.682562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:57.682562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:57.682562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:57.682562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:57.682562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:57.682562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:57.682562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:57.682562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:57.682562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:57.682562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:57.682562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:57.682562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:57.682562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:57.682562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:57.682562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:57.682562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:57.683997 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:57.683997 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:57.683997 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:57.683997 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 366/728] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:57.683997 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684cbf22-41fb-4389-9ccd-71e54f99140c None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:57.684285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684cbf22-41fb-4389-9ccd-71e54f99140c None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:57.684569 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684cbf22-41fb-4389-9ccd-71e54f99140c None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:57.685836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 367/729] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:57 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:57.690873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5cc83d-11d2-4edf-b78e-be42cf3e51f2 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:57.691181 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5cc83d-11d2-4edf-b78e-be42cf3e51f2 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:57.691456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5cc83d-11d2-4edf-b78e-be42cf3e51f2 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:57.695126 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 368/730] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:57 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:57.695126 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5cc83d-11d2-4edf-b78e-be42cf3e51f2 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:57.699437 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff5f54d-76bd-4170-a5c7-984af67b66ff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:57.700882 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff5f54d-76bd-4170-a5c7-984af67b66ff None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:57.715742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-cff5f54d-76bd-4170-a5c7-984af67b66ff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:57.715742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:57.715742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:57.715742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:57.715742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:57.715742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:57.715742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:57.715742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:57.715742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:57.715742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:57.715742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:57.715742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:57.715742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:57.715742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:57.715742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:57.715742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:57.716744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:57.716744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:57.716744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:57.716744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:57.716744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:57.716744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:57.716744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:57.716744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:57.716744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:57.716744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:57.716744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:57.716744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:57.716744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:57.716744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:57.716744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:57.716744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:57.716744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:57.717647 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:57.717647 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:57.717647 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:57.722049 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 363/731] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:57.723324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e279576-4569-41de-ad80-3c8b09e5e644 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:57.723611 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e279576-4569-41de-ad80-3c8b09e5e644 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:57.723886 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e279576-4569-41de-ad80-3c8b09e5e644 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:57.742108 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1e279576-4569-41de-ad80-3c8b09e5e644 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:57.743312 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1e279576-4569-41de-ad80-3c8b09e5e644 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:57.806818 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5cc83d-11d2-4edf-b78e-be42cf3e51f2 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:57.817274 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 364/732] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:57.819410 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa4f37f-d410-46c6-9ebd-431b6f553551 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:57.819692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa4f37f-d410-46c6-9ebd-431b6f553551 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:57.821121 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa4f37f-d410-46c6-9ebd-431b6f553551 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:57.841636 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7aa4f37f-d410-46c6-9ebd-431b6f553551 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:57.842823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7aa4f37f-d410-46c6-9ebd-431b6f553551 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:57.855122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ce4c64-dbdf-4264-a475-b75d40dcda34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:57.856048 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ce4c64-dbdf-4264-a475-b75d40dcda34 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:57.856374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ce4c64-dbdf-4264-a475-b75d40dcda34 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:57.856716 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ce4c64-dbdf-4264-a475-b75d40dcda34 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:57.857427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ce4c64-dbdf-4264-a475-b75d40dcda34 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:57.857801 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ce4c64-dbdf-4264-a475-b75d40dcda34 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:57.859341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ce4c64-dbdf-4264-a475-b75d40dcda34 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:57.874613 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 369/733] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:57.875711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ce4c64-dbdf-4264-a475-b75d40dcda34 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:57.879702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:57.881157 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:57.922355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 365/734] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:57.927417 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa4f37f-d410-46c6-9ebd-431b6f553551 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:58.039770 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ce4c64-dbdf-4264-a475-b75d40dcda34 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:58.087073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14af0e44-877f-44bb-90f1-588c95589fe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:58.087782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14af0e44-877f-44bb-90f1-588c95589fe3 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:58.088132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14af0e44-877f-44bb-90f1-588c95589fe3 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:58.088616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14af0e44-877f-44bb-90f1-588c95589fe3 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:58.089505 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14af0e44-877f-44bb-90f1-588c95589fe3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:58.089888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14af0e44-877f-44bb-90f1-588c95589fe3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:58.091631 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14af0e44-877f-44bb-90f1-588c95589fe3 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:58.106083 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 370/735] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:58.111351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0caac018-621b-47af-8c6e-3679730424c1 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:58.111644 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0caac018-621b-47af-8c6e-3679730424c1 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:58.111916 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0caac018-621b-47af-8c6e-3679730424c1 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:58.113325 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 371/736] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:27:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:27:58.113902 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0caac018-621b-47af-8c6e-3679730424c1 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:58.128318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa4f37f-d410-46c6-9ebd-431b6f553551 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:58.169304 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238b9ff2-2366-4c53-a133-bea6676ab615 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:58.170095 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238b9ff2-2366-4c53-a133-bea6676ab615 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:58.170320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238b9ff2-2366-4c53-a133-bea6676ab615 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:58.170595 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238b9ff2-2366-4c53-a133-bea6676ab615 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:58.178479 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238b9ff2-2366-4c53-a133-bea6676ab615 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:58.179853 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238b9ff2-2366-4c53-a133-bea6676ab615 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:58.182733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:58.192772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-238b9ff2-2366-4c53-a133-bea6676ab615 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 16 03:27:58.192772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:58.192772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:58.192772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:58.192772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:58.192772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:58.192772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:58.192772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:58.192772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:58.192772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:58.192772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:58.192772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:58.192772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:27:58.192772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:27:58.192772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:27:58.192772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:27:58.193747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:58.193747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:58.193747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:27:58.193747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:58.193747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:27:58.193747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:58.193747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:58.193747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:58.193747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:58.193747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:58.193747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:58.193747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:58.193747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:58.193747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:58.193747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:58.193747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:58.193747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:58.193747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:58.194773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:58.194773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:58.194773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:27:58.194773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:27:58.194773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:27:58.194773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:27:58.194773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:27:58.194773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:27:58.194773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 16 03:27:58.194773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:58.194773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 366/737] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:27:57 2022] GET /identity/v3/users/swift => generated 83 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:58.195417 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238b9ff2-2366-4c53-a133-bea6676ab615 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:58.221696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:58.229599 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0caac018-621b-47af-8c6e-3679730424c1 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:58.264848 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5c1014-14e5-4212-ad4a-77f5d3671b5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:58.265483 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5c1014-14e5-4212-ad4a-77f5d3671b5a None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:58.265829 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5c1014-14e5-4212-ad4a-77f5d3671b5a None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:58.266130 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5c1014-14e5-4212-ad4a-77f5d3671b5a None admin] PATH_INFO: `/v3/users/nova` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:58.273744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5c1014-14e5-4212-ad4a-77f5d3671b5a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:58.275245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5c1014-14e5-4212-ad4a-77f5d3671b5a None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:58.288400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-5f5c1014-14e5-4212-ad4a-77f5d3671b5a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 16 03:27:58.288400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:58.288400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:58.288400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:58.288400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:58.288400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:58.288400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:58.288400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:58.288400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:58.288400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:58.288400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:58.288400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:58.288400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:27:58.288400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:27:58.288400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:27:58.288400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:27:58.289521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:58.289521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:58.289521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:27:58.289521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:58.289521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:27:58.289521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:27:58.289521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:58.289521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:58.289521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:58.289521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:58.289521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:58.289521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:58.289521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:58.289521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:58.289521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:58.289521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:58.289521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:58.289521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:58.290487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:58.290487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:58.290487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:27:58.290487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:27:58.290487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:27:58.290487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:27:58.290487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:27:58.290487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:27:58.290487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 16 03:27:58.290487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:58.290487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 372/738] 10.222.0.49 () {56 vars in 1093 bytes} [Fri Sep 16 03:27:58 2022] GET /identity/v3/users/nova => generated 82 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:58.291141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debe9b3b-f440-469a-8a23-5a1579182d43 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:58.291141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debe9b3b-f440-469a-8a23-5a1579182d43 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:58.291613 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debe9b3b-f440-469a-8a23-5a1579182d43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:58.311323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-debe9b3b-f440-469a-8a23-5a1579182d43 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:27:58.312535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-debe9b3b-f440-469a-8a23-5a1579182d43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:27:58.383613 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 373/739] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:27:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 03:27:58.384756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debe9b3b-f440-469a-8a23-5a1579182d43 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:58.411024 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238b9ff2-2366-4c53-a133-bea6676ab615 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:58.440831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:58.448955 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf84422-d768-4209-b623-91d7bd5cb578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:58.449569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf84422-d768-4209-b623-91d7bd5cb578 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:58.449840 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf84422-d768-4209-b623-91d7bd5cb578 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:58.450112 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf84422-d768-4209-b623-91d7bd5cb578 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:58.450750 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf84422-d768-4209-b623-91d7bd5cb578 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:58.452148 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf84422-d768-4209-b623-91d7bd5cb578 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:58.487631 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debe9b3b-f440-469a-8a23-5a1579182d43 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:58.521499 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcfd88c-6a27-40ae-b218-dd49a46b98f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:58.522114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcfd88c-6a27-40ae-b218-dd49a46b98f4 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:58.522414 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcfd88c-6a27-40ae-b218-dd49a46b98f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:58.522668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcfd88c-6a27-40ae-b218-dd49a46b98f4 None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:58.523424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dcfd88c-6a27-40ae-b218-dd49a46b98f4 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:58.523728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dcfd88c-6a27-40ae-b218-dd49a46b98f4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:58.525150 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dcfd88c-6a27-40ae-b218-dd49a46b98f4 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:58.542369 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 374/740] 10.222.0.49 () {56 vars in 1135 bytes} [Fri Sep 16 03:27:58 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:58.542950 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcfd88c-6a27-40ae-b218-dd49a46b98f4 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:58.578528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 367/741] 10.222.0.49 () {60 vars in 1141 bytes} [Fri Sep 16 03:27:58 2022] POST /identity/v3/services => generated 240 bytes in 378 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:27:58.578528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf84422-d768-4209-b623-91d7bd5cb578 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:58.678354 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcfd88c-6a27-40ae-b218-dd49a46b98f4 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:58.684583 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:58.695705 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:58.719849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf84422-d768-4209-b623-91d7bd5cb578 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:58.726606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9fa5da-8e19-4dae-add0-9f1236bb1d21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:58.727294 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9fa5da-8e19-4dae-add0-9f1236bb1d21 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:58.727579 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9fa5da-8e19-4dae-add0-9f1236bb1d21 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:58.727859 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9fa5da-8e19-4dae-add0-9f1236bb1d21 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:58.740467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9fa5da-8e19-4dae-add0-9f1236bb1d21 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:27:58.745418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-9a9fa5da-8e19-4dae-add0-9f1236bb1d21 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:58.745418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:58.745418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:58.745418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:58.745418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:58.745418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:58.745418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:58.745418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:58.745418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:58.745418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:58.745418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:58.745418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:58.745418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:27:58.745418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:27:58.745418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:27:58.745418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:27:58.746430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:27:58.746430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:27:58.746430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:27:58.746430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:27:58.746430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:58.746430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:58.746430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:58.746430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:58.746430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:58.746430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:58.746430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:58.746430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:58.746430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:58.746430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:58.746430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:58.746430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:58.747410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:58.747410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:58.747410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:58.747410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:58.747410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:27:58.747410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:27:58.747410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:27:58.747410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:27:58.747410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:27:58.747410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:27:58.747410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:27:58.747410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:58.747410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 375/742] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:27:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:58.748129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9fa5da-8e19-4dae-add0-9f1236bb1d21 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:58.763178 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9976cb8-cb36-4ea9-960a-b498a18e0372 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:58.764053 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9976cb8-cb36-4ea9-960a-b498a18e0372 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:58.764335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9976cb8-cb36-4ea9-960a-b498a18e0372 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:58.764638 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9976cb8-cb36-4ea9-960a-b498a18e0372 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:58.765352 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9976cb8-cb36-4ea9-960a-b498a18e0372 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:58.765662 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9976cb8-cb36-4ea9-960a-b498a18e0372 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:58.767743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9976cb8-cb36-4ea9-960a-b498a18e0372 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:58.787708 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 368/743] 10.222.0.49 () {56 vars in 1133 bytes} [Fri Sep 16 03:27:58 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:58.789180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9976cb8-cb36-4ea9-960a-b498a18e0372 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:58.871473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9fa5da-8e19-4dae-add0-9f1236bb1d21 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:58.909496 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4442240-7557-4e53-b53e-bc02a04fc079 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:58.910379 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4442240-7557-4e53-b53e-bc02a04fc079 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:58.910735 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4442240-7557-4e53-b53e-bc02a04fc079 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:58.911495 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4442240-7557-4e53-b53e-bc02a04fc079 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:58.920254 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9976cb8-cb36-4ea9-960a-b498a18e0372 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:58.956760 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:58.960892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8c6968-3324-4cc7-bdda-51e958bbded4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:58.961514 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8c6968-3324-4cc7-bdda-51e958bbded4 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:58.961797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8c6968-3324-4cc7-bdda-51e958bbded4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:58.962074 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8c6968-3324-4cc7-bdda-51e958bbded4 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:58.963225 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8c6968-3324-4cc7-bdda-51e958bbded4 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:58.964970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8c6968-3324-4cc7-bdda-51e958bbded4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:58.970641 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:58.995922 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4442240-7557-4e53-b53e-bc02a04fc079 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:58.997396 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4442240-7557-4e53-b53e-bc02a04fc079 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:59.057908 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 369/744] 10.222.0.49 () {60 vars in 1141 bytes} [Fri Sep 16 03:27:58 2022] POST /identity/v3/services => generated 251 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:27:59.059187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8c6968-3324-4cc7-bdda-51e958bbded4 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:59.181481 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 376/745] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:27:58 2022] GET /identity/v3/auth/tokens => generated 898 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 03:27:59.182475 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4442240-7557-4e53-b53e-bc02a04fc079 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:59.206354 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8c6968-3324-4cc7-bdda-51e958bbded4 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:59.269607 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff2fb5b-2342-419e-885e-f415ac2bda0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:59.271727 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff2fb5b-2342-419e-885e-f415ac2bda0c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:59.272017 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff2fb5b-2342-419e-885e-f415ac2bda0c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:59.272294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff2fb5b-2342-419e-885e-f415ac2bda0c None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:59.272971 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff2fb5b-2342-419e-885e-f415ac2bda0c None admin] RBAC: Adding query filter params (name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:59.273274 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff2fb5b-2342-419e-885e-f415ac2bda0c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:59.274650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff2fb5b-2342-419e-885e-f415ac2bda0c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:59.286637 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 370/746] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:27:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:59.287556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff2fb5b-2342-419e-885e-f415ac2bda0c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:59.332358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:59.356627 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:59.383119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4442240-7557-4e53-b53e-bc02a04fc079 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:59.413183 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff2fb5b-2342-419e-885e-f415ac2bda0c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:59.421916 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94d05a1-9f96-4eff-aff1-f8ba52844eec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:59.423350 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94d05a1-9f96-4eff-aff1-f8ba52844eec None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:59.423718 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94d05a1-9f96-4eff-aff1-f8ba52844eec None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:59.424062 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94d05a1-9f96-4eff-aff1-f8ba52844eec None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:59.434638 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94d05a1-9f96-4eff-aff1-f8ba52844eec None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:59.436332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94d05a1-9f96-4eff-aff1-f8ba52844eec None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:59.452290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-d94d05a1-9f96-4eff-aff1-f8ba52844eec None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:59.452290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:59.452290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:59.452290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:59.452290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:59.452290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:59.452290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:59.452290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:59.452290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:59.452290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:59.452290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:59.452290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:59.452290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:59.452290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:59.452290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:59.452290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:59.454405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:59.454405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:59.454405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:59.454405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:59.454405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:59.454405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:59.454405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:59.454405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:59.454405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:59.454405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:59.454405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:59.454405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:59.454405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:59.454405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:59.454405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:59.454405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:59.454405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:59.455604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:59.455604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:59.455604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:59.455604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 377/747] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:59.460675 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94d05a1-9f96-4eff-aff1-f8ba52844eec None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:59.461331 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2374fcf-56ed-4c98-93b5-90aaac8bd9df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:59.462141 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2374fcf-56ed-4c98-93b5-90aaac8bd9df None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:59.462558 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2374fcf-56ed-4c98-93b5-90aaac8bd9df None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:59.463016 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2374fcf-56ed-4c98-93b5-90aaac8bd9df None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:59.478086 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2374fcf-56ed-4c98-93b5-90aaac8bd9df None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:59.480574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2374fcf-56ed-4c98-93b5-90aaac8bd9df None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:59.500201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-b2374fcf-56ed-4c98-93b5-90aaac8bd9df None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:59.500201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:59.500201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:59.500201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:59.500201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:59.500201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:59.500201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:59.500201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:59.500201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:59.500201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:59.500201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:59.500201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:59.500201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:59.500201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:59.500201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:59.500201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:59.502063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:59.502063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:59.502063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:59.502063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:59.502063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:59.502063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:59.502063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:59.502063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:59.502063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:59.502063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:59.502063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:59.502063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:59.502063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:59.502063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:59.502063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:59.502063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:59.502063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:59.503198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:59.503198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:59.503198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:27:59.504381 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 371/748] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:59.505339 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2374fcf-56ed-4c98-93b5-90aaac8bd9df None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:59.593320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94d05a1-9f96-4eff-aff1-f8ba52844eec None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:59.611703 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:59.627415 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2374fcf-56ed-4c98-93b5-90aaac8bd9df None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:59.632521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7da28e-4d31-4eb7-96e0-3651ab5e1bd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:59.633360 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7da28e-4d31-4eb7-96e0-3651ab5e1bd5 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:59.633810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7da28e-4d31-4eb7-96e0-3651ab5e1bd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:59.634221 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7da28e-4d31-4eb7-96e0-3651ab5e1bd5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:59.636422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:59.650104 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7da28e-4d31-4eb7-96e0-3651ab5e1bd5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:59.652337 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7da28e-4d31-4eb7-96e0-3651ab5e1bd5 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:59.668525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-2a7da28e-4d31-4eb7-96e0-3651ab5e1bd5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:59.668525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:27:59.668525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:27:59.668525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:27:59.668525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:27:59.668525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:27:59.668525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:27:59.668525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:27:59.668525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:27:59.668525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:27:59.668525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:27:59.668525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:27:59.668525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:27:59.668525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:27:59.668525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:27:59.668525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:27:59.669694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:27:59.669694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:27:59.669694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:27:59.669694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:27:59.669694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:27:59.669694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:27:59.669694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:27:59.669694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:27:59.669694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:27:59.669694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:27:59.669694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:27:59.669694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:27:59.669694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:27:59.669694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:27:59.669694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:27:59.669694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:27:59.669694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:27:59.670657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:27:59.670657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:27:59.670657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:27:59.671297 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 378/749] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:27:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:27:59.672701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7da28e-4d31-4eb7-96e0-3651ab5e1bd5 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:59.675370 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfddb0d-6e48-4c5c-87b2-b775dcbab249 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:59.676483 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfddb0d-6e48-4c5c-87b2-b775dcbab249 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:59.677929 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfddb0d-6e48-4c5c-87b2-b775dcbab249 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:59.678385 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfddb0d-6e48-4c5c-87b2-b775dcbab249 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:59.679280 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfddb0d-6e48-4c5c-87b2-b775dcbab249 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:59.679780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfddb0d-6e48-4c5c-87b2-b775dcbab249 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:59.681459 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfddb0d-6e48-4c5c-87b2-b775dcbab249 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:59.697820 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 372/750] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:59.698971 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfddb0d-6e48-4c5c-87b2-b775dcbab249 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:59.782516 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7da28e-4d31-4eb7-96e0-3651ab5e1bd5 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:59.822933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e887a86-19a9-4259-aeb4-fa8f3b17d684 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:59.824502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e887a86-19a9-4259-aeb4-fa8f3b17d684 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:59.824970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e887a86-19a9-4259-aeb4-fa8f3b17d684 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:59.825445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e887a86-19a9-4259-aeb4-fa8f3b17d684 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:59.826866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e887a86-19a9-4259-aeb4-fa8f3b17d684 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:59.827392 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e887a86-19a9-4259-aeb4-fa8f3b17d684 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:59.830154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e887a86-19a9-4259-aeb4-fa8f3b17d684 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:59.832764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfddb0d-6e48-4c5c-87b2-b775dcbab249 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:27:59.847300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 379/751] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:59.849161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e887a86-19a9-4259-aeb4-fa8f3b17d684 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:59.869217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf86075-c340-4ad8-a91d-2f4ce861aba8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:27:59.869873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf86075-c340-4ad8-a91d-2f4ce861aba8 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:27:59.870094 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf86075-c340-4ad8-a91d-2f4ce861aba8 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:27:59.870390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf86075-c340-4ad8-a91d-2f4ce861aba8 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:27:59.871463 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf86075-c340-4ad8-a91d-2f4ce861aba8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:27:59.872035 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf86075-c340-4ad8-a91d-2f4ce861aba8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:27:59.873454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf86075-c340-4ad8-a91d-2f4ce861aba8 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:27:59.887900 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 373/752] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:27:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:27:59.888804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf86075-c340-4ad8-a91d-2f4ce861aba8 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:27:59.889452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:59.928526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:27:59.976350 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e887a86-19a9-4259-aeb4-fa8f3b17d684 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:00.018190 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1862a8a-ab87-4930-bb8e-c39d105cf9c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:00.018823 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1862a8a-ab87-4930-bb8e-c39d105cf9c7 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:00.019121 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1862a8a-ab87-4930-bb8e-c39d105cf9c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:00.019389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1862a8a-ab87-4930-bb8e-c39d105cf9c7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:00.028612 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1862a8a-ab87-4930-bb8e-c39d105cf9c7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:00.030336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1862a8a-ab87-4930-bb8e-c39d105cf9c7 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:00.034366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf86075-c340-4ad8-a91d-2f4ce861aba8 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:00.042648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-e1862a8a-ab87-4930-bb8e-c39d105cf9c7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:00.042648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:00.042648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:00.042648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:00.042648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:00.042648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:00.042648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:00.042648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:00.042648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:00.042648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:00.042648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:00.042648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:00.042648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:28:00.042648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:28:00.042648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:28:00.043824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:28:00.043824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:00.043824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:00.043824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:00.043824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:00.043824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:00.043824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:00.043824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:00.043824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:00.043824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:00.043824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:00.043824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:00.043824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:00.043824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:00.043824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:00.043824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:00.043824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:00.044745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:28:00.044745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:28:00.044745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:28:00.044745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:28:00.044745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:28:00.044745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:28:00.044745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:00.044745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:00.045198 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 380/753] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:00.046220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1862a8a-ab87-4930-bb8e-c39d105cf9c7 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:00.075685 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3a9af3-60ed-4326-abb4-5990e98c2771 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:00.076663 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3a9af3-60ed-4326-abb4-5990e98c2771 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:00.077096 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3a9af3-60ed-4326-abb4-5990e98c2771 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:00.077605 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3a9af3-60ed-4326-abb4-5990e98c2771 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:00.091305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3a9af3-60ed-4326-abb4-5990e98c2771 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:00.097716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3a9af3-60ed-4326-abb4-5990e98c2771 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:00.113704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-ed3a9af3-60ed-4326-abb4-5990e98c2771 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:00.113704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:00.113704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:00.113704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:00.113704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:00.113704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:00.113704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:00.113704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:00.113704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:00.113704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:00.113704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:00.113704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:00.113704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:28:00.113704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:28:00.113704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:28:00.116679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:28:00.116679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:00.116679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:00.116679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:00.116679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:00.116679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:00.116679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:00.116679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:00.116679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:00.116679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:00.116679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:00.116679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:00.116679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:00.116679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:00.116679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:00.116679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:00.116679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:00.117630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:28:00.117630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:28:00.117630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:28:00.117630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:28:00.117630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:28:00.117630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:28:00.117630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:00.117630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:00.127048 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 374/754] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:27:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:00.128106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3a9af3-60ed-4326-abb4-5990e98c2771 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:00.165747 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:00.176996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1862a8a-ab87-4930-bb8e-c39d105cf9c7 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:00.220133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f7663d-5446-48d2-9d72-03207f162fd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:00.221046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f7663d-5446-48d2-9d72-03207f162fd5 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:00.221391 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f7663d-5446-48d2-9d72-03207f162fd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:00.222103 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f7663d-5446-48d2-9d72-03207f162fd5 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:00.227288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:00.235017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f7663d-5446-48d2-9d72-03207f162fd5 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:00.236714 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f7663d-5446-48d2-9d72-03207f162fd5 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:00.245689 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3a9af3-60ed-4326-abb4-5990e98c2771 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:00.248811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-97f7663d-5446-48d2-9d72-03207f162fd5 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 03:28:00.248811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:00.248811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:00.248811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:00.248811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:00.248811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:00.248811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:00.248811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:00.248811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:00.248811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:00.248811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:00.248811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:00.248811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:28:00.248811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:28:00.248811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:28:00.248811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:28:00.249863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:00.249863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:00.249863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:28:00.249863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:00.249863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:28:00.249863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:00.249863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:00.249863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:00.249863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:00.249863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:00.249863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:00.249863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:00.249863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:00.249863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:00.249863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:00.249863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:00.249863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:00.249863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:00.251153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:00.251153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:00.251153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:28:00.251153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:28:00.251153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:28:00.251153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:28:00.251153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:28:00.251153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:28:00.251153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 03:28:00.251153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:00.253283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 381/755] 10.222.0.49 () {56 vars in 1117 bytes} [Fri Sep 16 03:28:00 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:00.254281 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f7663d-5446-48d2-9d72-03207f162fd5 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:00.287885 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b7c831-9bd2-45aa-9794-14eeb1629733 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:00.288494 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b7c831-9bd2-45aa-9794-14eeb1629733 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:00.288769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b7c831-9bd2-45aa-9794-14eeb1629733 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:00.289037 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b7c831-9bd2-45aa-9794-14eeb1629733 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:00.289716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b7c831-9bd2-45aa-9794-14eeb1629733 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:00.290019 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b7c831-9bd2-45aa-9794-14eeb1629733 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:00.292183 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b7c831-9bd2-45aa-9794-14eeb1629733 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:00.314989 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 375/756] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:28:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:00.316226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b7c831-9bd2-45aa-9794-14eeb1629733 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:00.390378 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f7663d-5446-48d2-9d72-03207f162fd5 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:00.427351 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b7c831-9bd2-45aa-9794-14eeb1629733 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:00.432731 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866340b3-fd81-45a5-a923-5dc51a0d65ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:00.433360 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866340b3-fd81-45a5-a923-5dc51a0d65ba None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:00.433675 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866340b3-fd81-45a5-a923-5dc51a0d65ba None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:00.433924 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866340b3-fd81-45a5-a923-5dc51a0d65ba None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:00.434605 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866340b3-fd81-45a5-a923-5dc51a0d65ba None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:00.435479 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866340b3-fd81-45a5-a923-5dc51a0d65ba None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:00.437400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866340b3-fd81-45a5-a923-5dc51a0d65ba None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:00.460257 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 382/757] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:28:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:00.461254 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866340b3-fd81-45a5-a923-5dc51a0d65ba None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:00.473069 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506693cc-0cce-46bc-8fca-83d4f1b3c057 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:00.473993 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506693cc-0cce-46bc-8fca-83d4f1b3c057 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:00.475108 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506693cc-0cce-46bc-8fca-83d4f1b3c057 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:00.475545 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506693cc-0cce-46bc-8fca-83d4f1b3c057 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:00.476438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506693cc-0cce-46bc-8fca-83d4f1b3c057 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:00.477408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506693cc-0cce-46bc-8fca-83d4f1b3c057 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:00.479198 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:00.481178 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506693cc-0cce-46bc-8fca-83d4f1b3c057 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:00.504755 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 376/758] 10.222.0.49 () {56 vars in 1149 bytes} [Fri Sep 16 03:28:00 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:00.505840 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506693cc-0cce-46bc-8fca-83d4f1b3c057 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:00.509723 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:00.589391 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866340b3-fd81-45a5-a923-5dc51a0d65ba None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:00.617293 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506693cc-0cce-46bc-8fca-83d4f1b3c057 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:00.637434 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8a6336-cd31-44e5-a0ff-34e4d473e6df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:00.638230 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8a6336-cd31-44e5-a0ff-34e4d473e6df None admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:00.638612 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8a6336-cd31-44e5-a0ff-34e4d473e6df None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:00.639018 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8a6336-cd31-44e5-a0ff-34e4d473e6df None admin] PATH_INFO: `/v3/projects/0bb7350474f349d6aee197ff35536167/users/ded1bcd10e62499a9aaa0ece5aa55c80/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:00.642097 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8a6336-cd31-44e5-a0ff-34e4d473e6df None admin] RBAC: Authorizing `identity:create_grant(project_id=0bb7350474f349d6aee197ff35536167, user_id=ded1bcd10e62499a9aaa0ece5aa55c80, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:00.644270 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8a6336-cd31-44e5-a0ff-34e4d473e6df None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:00.654695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d07d4ec-af02-4a5e-b7f7-f404154d7631 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:00.655583 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d07d4ec-af02-4a5e-b7f7-f404154d7631 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:00.655890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d07d4ec-af02-4a5e-b7f7-f404154d7631 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:00.656165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d07d4ec-af02-4a5e-b7f7-f404154d7631 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:00.657528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d07d4ec-af02-4a5e-b7f7-f404154d7631 None admin] RBAC: Adding query filter params (role.id=89c4b3a971224d468d33d2dc411230a2, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=aa30da9972e140d7905c522576681429) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:00.657851 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d07d4ec-af02-4a5e-b7f7-f404154d7631 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:00.659295 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d07d4ec-af02-4a5e-b7f7-f404154d7631 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:00.665010 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 377/759] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:28:00 2022] GET /identity/v3/role_assignments?user.id=aa30da9972e140d7905c522576681429&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=89c4b3a971224d468d33d2dc411230a2 => generated 259 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:00.665636 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d07d4ec-af02-4a5e-b7f7-f404154d7631 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:00.710169 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 383/760] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:28:00 2022] PUT /identity/v3/projects/0bb7350474f349d6aee197ff35536167/users/ded1bcd10e62499a9aaa0ece5aa55c80/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:28:00.764474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:00.805318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d07d4ec-af02-4a5e-b7f7-f404154d7631 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:00.845430 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab196eec-fec4-44c3-9671-100af2f78fe9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:00.846073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab196eec-fec4-44c3-9671-100af2f78fe9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:00.846359 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab196eec-fec4-44c3-9671-100af2f78fe9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:00.846635 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab196eec-fec4-44c3-9671-100af2f78fe9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:00.864900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dcba64-85b3-4685-81e8-a7e089df608d None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:00.865569 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dcba64-85b3-4685-81e8-a7e089df608d None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:00.866094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dcba64-85b3-4685-81e8-a7e089df608d None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:00.868341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 384/761] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:00.874741 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e450d4e-ed9f-4db3-a467-30de06de0a1b None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:00.875096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e450d4e-ed9f-4db3-a467-30de06de0a1b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:00.875402 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e450d4e-ed9f-4db3-a467-30de06de0a1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:00.910474 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-8e450d4e-ed9f-4db3-a467-30de06de0a1b None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:00.911652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-8e450d4e-ed9f-4db3-a467-30de06de0a1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:00.950353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab196eec-fec4-44c3-9671-100af2f78fe9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:00.952039 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab196eec-fec4-44c3-9671-100af2f78fe9 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:01.016327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:01.025157 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 385/762] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:00 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:01.045164 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:01.082968 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 378/763] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:28:00 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 03:28:01.094496 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e450d4e-ed9f-4db3-a467-30de06de0a1b None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:01.198684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e450d4e-ed9f-4db3-a467-30de06de0a1b None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:01.242416 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd02574-aeb5-4394-b0f7-dc100e02aee8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:01.243088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd02574-aeb5-4394-b0f7-dc100e02aee8 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:01.243373 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd02574-aeb5-4394-b0f7-dc100e02aee8 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:01.243645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd02574-aeb5-4394-b0f7-dc100e02aee8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:01.255427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd02574-aeb5-4394-b0f7-dc100e02aee8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:01.256891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd02574-aeb5-4394-b0f7-dc100e02aee8 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:01.283045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-0fd02574-aeb5-4394-b0f7-dc100e02aee8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:01.283045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:01.283045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:01.283045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:01.283045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:01.283045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:01.283045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:01.283045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:01.283045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:01.283045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:01.283045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:01.283045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:01.283045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:01.283045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:01.283045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:01.283045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:01.284070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:01.284070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:01.284070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:01.284070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:01.284070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:01.284070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:01.284070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:01.284070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:01.284070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:01.284070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:01.284070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:01.284070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:01.284070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:01.284070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:01.284070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:01.284070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:01.284070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:01.284979 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:01.284979 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:01.284979 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:01.284979 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 386/764] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:01.290019 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab196eec-fec4-44c3-9671-100af2f78fe9 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:01.427859 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab196eec-fec4-44c3-9671-100af2f78fe9 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:01.448176 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd02574-aeb5-4394-b0f7-dc100e02aee8 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:01.471140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e2df1b-e10d-4e7e-a2ba-821491db4c52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:01.472088 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e2df1b-e10d-4e7e-a2ba-821491db4c52 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:01.472571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e2df1b-e10d-4e7e-a2ba-821491db4c52 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:01.473070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e2df1b-e10d-4e7e-a2ba-821491db4c52 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:01.473913 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e2df1b-e10d-4e7e-a2ba-821491db4c52 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:01.474417 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e2df1b-e10d-4e7e-a2ba-821491db4c52 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:01.476078 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e2df1b-e10d-4e7e-a2ba-821491db4c52 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:01.495754 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 379/765] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:01.503085 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e2df1b-e10d-4e7e-a2ba-821491db4c52 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:01.512899 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:01.556001 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:01.594541 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd02574-aeb5-4394-b0f7-dc100e02aee8 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:01.632398 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d32045-f910-49c7-ac81-7a202f5ef5e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:01.633501 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d32045-f910-49c7-ac81-7a202f5ef5e4 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:01.634016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d32045-f910-49c7-ac81-7a202f5ef5e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:01.634512 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d32045-f910-49c7-ac81-7a202f5ef5e4 None admin] PATH_INFO: `/v3/services/network` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:01.635327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d32045-f910-49c7-ac81-7a202f5ef5e4 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:01.637120 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d32045-f910-49c7-ac81-7a202f5ef5e4 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:01.646268 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e2df1b-e10d-4e7e-a2ba-821491db4c52 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:01.648088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-a3d32045-f910-49c7-ac81-7a202f5ef5e4 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 16 03:28:01.648088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:01.648088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:01.648088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:01.648088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:01.648088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:01.648088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:01.648088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:01.648088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 16 03:28:01.648088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:01.648088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:01.648088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:01.648088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:01.648088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:01.648088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:01.648088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:01.648088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:01.649051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:01.649051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:01.649051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:01.649051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:01.649051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:01.649051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:01.649051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:01.649051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:01.649051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:01.649051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:01.649051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:01.649051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:01.649051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:01.649051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:01.649051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:01.649051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:01.649919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:01.649919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:01.649919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:01.649919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:01.649919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:01.649919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:01.649919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:01.649919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:01.649919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:01.649919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:01.649919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:01.649919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:01.649919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 16 03:28:01.649919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:01.652736 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 387/766] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:28:01 2022] GET /identity/v3/services/network => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:01.659844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523b65f3-884a-4688-90f0-3ef983c61566 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:01.660533 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523b65f3-884a-4688-90f0-3ef983c61566 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:01.661028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523b65f3-884a-4688-90f0-3ef983c61566 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:01.662842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 388/767] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:01 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:01.663900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523b65f3-884a-4688-90f0-3ef983c61566 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:01.684126 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79df7109-c50a-4504-aae7-eeff5bb101f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:01.684807 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79df7109-c50a-4504-aae7-eeff5bb101f2 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:01.685068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79df7109-c50a-4504-aae7-eeff5bb101f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:01.685348 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79df7109-c50a-4504-aae7-eeff5bb101f2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:01.694069 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79df7109-c50a-4504-aae7-eeff5bb101f2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:01.695670 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79df7109-c50a-4504-aae7-eeff5bb101f2 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:01.709911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-79df7109-c50a-4504-aae7-eeff5bb101f2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:01.709911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:01.709911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:01.709911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:01.709911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:01.709911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:01.709911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:01.709911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:01.709911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:01.709911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:01.709911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:01.709911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:01.709911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:28:01.709911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:28:01.709911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:28:01.710893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:28:01.710893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:01.710893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:01.710893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:01.710893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:01.710893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:01.710893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:01.710893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:01.710893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:01.710893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:01.710893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:01.710893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:01.710893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:01.710893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:01.710893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:01.710893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:01.710893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:01.712146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:28:01.712146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:28:01.712146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:28:01.712146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:28:01.712146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:28:01.712146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:28:01.712146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:01.712146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:01.712146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 380/768] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:28:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:01.712638 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ae2df0-27ac-4ce8-acc9-54c5e685b542 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:01.717547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ae2df0-27ac-4ce8-acc9-54c5e685b542 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:01.717547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ae2df0-27ac-4ce8-acc9-54c5e685b542 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:01.735673 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-63ae2df0-27ac-4ce8-acc9-54c5e685b542 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:01.736777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-63ae2df0-27ac-4ce8-acc9-54c5e685b542 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:01.793207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523b65f3-884a-4688-90f0-3ef983c61566 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:01.811965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 381/769] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:01 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:01.813120 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ae2df0-27ac-4ce8-acc9-54c5e685b542 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:01.813445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:01.835691 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:01.837315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135c0556-b49c-4068-a239-7ef1c010b985 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:01.838227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135c0556-b49c-4068-a239-7ef1c010b985 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:01.839133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135c0556-b49c-4068-a239-7ef1c010b985 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:01.839613 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135c0556-b49c-4068-a239-7ef1c010b985 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:01.840496 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135c0556-b49c-4068-a239-7ef1c010b985 None admin] RBAC: Adding query filter params (name=network) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:01.841080 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135c0556-b49c-4068-a239-7ef1c010b985 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:01.842724 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135c0556-b49c-4068-a239-7ef1c010b985 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:01.849449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 389/770] 10.222.0.49 () {56 vars in 1112 bytes} [Fri Sep 16 03:28:01 2022] GET /identity/v3/services?name=network => generated 124 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:01.855244 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135c0556-b49c-4068-a239-7ef1c010b985 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:01.925229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ae2df0-27ac-4ce8-acc9-54c5e685b542 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:01.967759 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112b6c8a-aa4e-4d93-bfff-2e7019b735a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:01.969612 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112b6c8a-aa4e-4d93-bfff-2e7019b735a7 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:01.969981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112b6c8a-aa4e-4d93-bfff-2e7019b735a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:01.970493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112b6c8a-aa4e-4d93-bfff-2e7019b735a7 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:01.971440 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112b6c8a-aa4e-4d93-bfff-2e7019b735a7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:01.971965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112b6c8a-aa4e-4d93-bfff-2e7019b735a7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:01.973524 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112b6c8a-aa4e-4d93-bfff-2e7019b735a7 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:01.991007 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 382/771] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:28:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:02.004870 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112b6c8a-aa4e-4d93-bfff-2e7019b735a7 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:02.032833 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135c0556-b49c-4068-a239-7ef1c010b985 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:02.080227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3315b0a9-7b30-4865-80dc-640fcdb0d743 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:02.081126 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3315b0a9-7b30-4865-80dc-640fcdb0d743 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:02.081593 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3315b0a9-7b30-4865-80dc-640fcdb0d743 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:02.082049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3315b0a9-7b30-4865-80dc-640fcdb0d743 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:02.082880 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3315b0a9-7b30-4865-80dc-640fcdb0d743 None admin] RBAC: Adding query filter params (type=network) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:02.083414 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3315b0a9-7b30-4865-80dc-640fcdb0d743 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:02.085021 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3315b0a9-7b30-4865-80dc-640fcdb0d743 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:02.096370 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 390/772] 10.222.0.49 () {56 vars in 1112 bytes} [Fri Sep 16 03:28:01 2022] GET /identity/v3/services?type=network => generated 350 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:02.101743 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3315b0a9-7b30-4865-80dc-640fcdb0d743 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:02.124942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:02.159834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112b6c8a-aa4e-4d93-bfff-2e7019b735a7 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:02.166616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:02.205581 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81254c19-3ad3-40c6-ac2d-f64f0f318ee1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:02.206194 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81254c19-3ad3-40c6-ac2d-f64f0f318ee1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:02.206471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81254c19-3ad3-40c6-ac2d-f64f0f318ee1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:02.206745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81254c19-3ad3-40c6-ac2d-f64f0f318ee1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:02.207990 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81254c19-3ad3-40c6-ac2d-f64f0f318ee1 None admin] RBAC: Adding query filter params (role.id=974d839067e9411aa4cb850cee33f6ae, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=d40810f61ead4c3da414194b325da80a) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:02.208297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81254c19-3ad3-40c6-ac2d-f64f0f318ee1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:02.209980 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81254c19-3ad3-40c6-ac2d-f64f0f318ee1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:02.218995 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 383/773] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:28:02 2022] GET /identity/v3/role_assignments?user.id=d40810f61ead4c3da414194b325da80a&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=974d839067e9411aa4cb850cee33f6ae => generated 259 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:02.220247 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81254c19-3ad3-40c6-ac2d-f64f0f318ee1 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:02.242272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3315b0a9-7b30-4865-80dc-640fcdb0d743 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:02.297425 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a5d03a-63e9-44b6-904a-0229014c9aec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:02.298655 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a5d03a-63e9-44b6-904a-0229014c9aec None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:02.299178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a5d03a-63e9-44b6-904a-0229014c9aec None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:02.299680 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a5d03a-63e9-44b6-904a-0229014c9aec None admin] PATH_INFO: `/v3/endpoints` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:02.300699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a5d03a-63e9-44b6-904a-0229014c9aec None admin] RBAC: Adding query filter params (interface=public, service_id=63b8727b33cb4e66b0718b4b4935829f, region_id=RegionOne) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:02.301226 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a5d03a-63e9-44b6-904a-0229014c9aec None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:02.302837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a5d03a-63e9-44b6-904a-0229014c9aec None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:02.319023 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 391/774] 10.222.0.49 () {56 vars in 1251 bytes} [Fri Sep 16 03:28:02 2022] GET /identity/v3/endpoints?service_id=63b8727b33cb4e66b0718b4b4935829f&interface=public®ion_id=RegionOne => generated 194 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:02.324391 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a5d03a-63e9-44b6-904a-0229014c9aec None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:02.420267 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81254c19-3ad3-40c6-ac2d-f64f0f318ee1 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:02.474289 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2444a22-5ecc-4a24-808a-dc22a5f59f32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:02.474959 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2444a22-5ecc-4a24-808a-dc22a5f59f32 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:02.475234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2444a22-5ecc-4a24-808a-dc22a5f59f32 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:02.475512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2444a22-5ecc-4a24-808a-dc22a5f59f32 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:02.476110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2444a22-5ecc-4a24-808a-dc22a5f59f32 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:02.477475 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2444a22-5ecc-4a24-808a-dc22a5f59f32 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:02.488107 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a5d03a-63e9-44b6-904a-0229014c9aec None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:02.494864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-e2444a22-5ecc-4a24-808a-dc22a5f59f32 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 16 03:28:02.494864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:02.494864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:02.494864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:02.494864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:02.494864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:02.494864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:02.494864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:02.494864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 16 03:28:02.494864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:02.494864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:02.494864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:02.494864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:02.494864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:02.494864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:02.494864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:02.494864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:02.498219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:02.498219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:02.498219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:02.498219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:02.498219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:02.498219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:02.498219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:02.498219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:02.498219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:02.498219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:02.498219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:02.498219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:02.498219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:02.498219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:02.498219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:02.498219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:02.499102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:02.499102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:02.499102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:02.499102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:02.499102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:02.499102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:02.499102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:02.499102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:02.499102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:02.499102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:02.499102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:02.499102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:02.499102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 16 03:28:02.499102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:02.499837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:02.503909 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 384/775] 10.222.0.49 () {56 vars in 1129 bytes} [Fri Sep 16 03:28:02 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:02.508489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2444a22-5ecc-4a24-808a-dc22a5f59f32 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:02.521861 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:02.538572 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66276b5f-74b8-41e5-88fe-8f3713d118a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:02.539920 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66276b5f-74b8-41e5-88fe-8f3713d118a8 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:02.540404 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66276b5f-74b8-41e5-88fe-8f3713d118a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:02.541006 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66276b5f-74b8-41e5-88fe-8f3713d118a8 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:02.545441 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66276b5f-74b8-41e5-88fe-8f3713d118a8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:02.545441 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66276b5f-74b8-41e5-88fe-8f3713d118a8 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:02.552287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 392/776] 10.222.0.49 () {56 vars in 1087 bytes} [Fri Sep 16 03:28:02 2022] GET /identity/v3/services => generated 772 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:02.637894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2444a22-5ecc-4a24-808a-dc22a5f59f32 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:02.693061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef303ba-df94-4b7b-9ede-6216e5057552 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:02.694095 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef303ba-df94-4b7b-9ede-6216e5057552 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:02.694577 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef303ba-df94-4b7b-9ede-6216e5057552 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:02.695101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef303ba-df94-4b7b-9ede-6216e5057552 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:02.696017 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef303ba-df94-4b7b-9ede-6216e5057552 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:02.696543 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef303ba-df94-4b7b-9ede-6216e5057552 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:02.698125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef303ba-df94-4b7b-9ede-6216e5057552 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:02.704245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 385/777] 10.222.0.49 () {56 vars in 1124 bytes} [Fri Sep 16 03:28:02 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:02.712022 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66276b5f-74b8-41e5-88fe-8f3713d118a8 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:02.840527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66276b5f-74b8-41e5-88fe-8f3713d118a8 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:02.879882 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f40754-15bf-4c34-9790-ae3c5648d176 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:02.880540 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f40754-15bf-4c34-9790-ae3c5648d176 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:02.880825 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f40754-15bf-4c34-9790-ae3c5648d176 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:02.881102 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f40754-15bf-4c34-9790-ae3c5648d176 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:02.881738 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f40754-15bf-4c34-9790-ae3c5648d176 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:02.882044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f40754-15bf-4c34-9790-ae3c5648d176 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:02.883952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f40754-15bf-4c34-9790-ae3c5648d176 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:02.891982 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 393/778] 10.222.0.49 () {56 vars in 1124 bytes} [Fri Sep 16 03:28:02 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:02.897913 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef303ba-df94-4b7b-9ede-6216e5057552 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:02.997853 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:03.044782 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef303ba-df94-4b7b-9ede-6216e5057552 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:03.096827 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e900cc-2e5b-48d4-96b9-6667f8e607f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:03.097416 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e900cc-2e5b-48d4-96b9-6667f8e607f0 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:03.097695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e900cc-2e5b-48d4-96b9-6667f8e607f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:03.097968 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e900cc-2e5b-48d4-96b9-6667f8e607f0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:03.098672 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e900cc-2e5b-48d4-96b9-6667f8e607f0 None admin] RBAC: Adding query filter params (interface=public, service_id=f76b7a46602742418b178be97fc41403, region_id=RegionOne) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:03.099002 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e900cc-2e5b-48d4-96b9-6667f8e607f0 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:03.100367 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e900cc-2e5b-48d4-96b9-6667f8e607f0 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:03.113824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 386/779] 10.222.0.49 () {56 vars in 1251 bytes} [Fri Sep 16 03:28:02 2022] GET /identity/v3/endpoints?service_id=f76b7a46602742418b178be97fc41403&interface=public®ion_id=RegionOne => generated 194 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:03.119403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f40754-15bf-4c34-9790-ae3c5648d176 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:03.186672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:03.275214 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f40754-15bf-4c34-9790-ae3c5648d176 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:03.338398 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704ff18d-e1b3-4dc3-bbb7-cffc3afbbef0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:03.339066 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704ff18d-e1b3-4dc3-bbb7-cffc3afbbef0 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:03.339352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704ff18d-e1b3-4dc3-bbb7-cffc3afbbef0 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:03.339627 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704ff18d-e1b3-4dc3-bbb7-cffc3afbbef0 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:03.340239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704ff18d-e1b3-4dc3-bbb7-cffc3afbbef0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:03.341616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704ff18d-e1b3-4dc3-bbb7-cffc3afbbef0 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:03.347512 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 394/780] 10.222.0.49 () {56 vars in 1087 bytes} [Fri Sep 16 03:28:03 2022] GET /identity/v3/services => generated 772 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:03.380789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66acac9-3de8-49b0-a7bf-2bb0d9e114b5 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:03.381101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66acac9-3de8-49b0-a7bf-2bb0d9e114b5 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:03.381350 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66acac9-3de8-49b0-a7bf-2bb0d9e114b5 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:03.383050 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 387/781] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:03.393295 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc35e022-d451-43c8-a1bb-9bde29c30d0b None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:03.393642 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc35e022-d451-43c8-a1bb-9bde29c30d0b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:03.393858 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc35e022-d451-43c8-a1bb-9bde29c30d0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:03.414097 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-fc35e022-d451-43c8-a1bb-9bde29c30d0b None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:03.415346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-fc35e022-d451-43c8-a1bb-9bde29c30d0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:03.494995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 395/782] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:03 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:03.639015 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d492e01-a6ba-41d8-ab79-b40c988c6f5a None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:03.639518 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d492e01-a6ba-41d8-ab79-b40c988c6f5a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:03.639732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d492e01-a6ba-41d8-ab79-b40c988c6f5a None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:03.641905 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 388/783] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:03.647324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ac0870-74d4-4b5f-99f5-8b426c906081 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:03.647747 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ac0870-74d4-4b5f-99f5-8b426c906081 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:03.647949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ac0870-74d4-4b5f-99f5-8b426c906081 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:03.667734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-f7ac0870-74d4-4b5f-99f5-8b426c906081 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:03.668921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-f7ac0870-74d4-4b5f-99f5-8b426c906081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:03.694607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:03.751573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 396/784] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:03 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:03.852956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d492e01-a6ba-41d8-ab79-b40c988c6f5a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:03.983181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d492e01-a6ba-41d8-ab79-b40c988c6f5a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:04.030785 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa6f70c-9493-4f0a-8228-413bd7f06664 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:04.031893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa6f70c-9493-4f0a-8228-413bd7f06664 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:04.032411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa6f70c-9493-4f0a-8228-413bd7f06664 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:04.032881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa6f70c-9493-4f0a-8228-413bd7f06664 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:04.038932 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:04.051738 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa6f70c-9493-4f0a-8228-413bd7f06664 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:28:04.056751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-1aa6f70c-9493-4f0a-8228-413bd7f06664 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:28:04.056751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:04.056751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:04.056751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:04.056751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:04.056751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:04.056751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:04.056751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:04.056751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:04.056751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:04.056751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:04.056751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:04.056751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:28:04.056751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:28:04.056751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:28:04.056751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:28:04.057656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:28:04.057656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:28:04.057656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:28:04.057656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:28:04.057656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:04.057656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:04.057656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:04.057656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:04.057656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:04.057656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:04.057656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:04.057656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:04.057656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:04.057656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:04.057656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:04.057656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:04.058507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:04.058507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:04.058507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:04.058507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:04.058507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:28:04.058507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:28:04.058507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:28:04.058507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:28:04.058507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:28:04.058507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:28:04.058507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:28:04.058507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:04.061067 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 389/785] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:28:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:04.066829 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ac0870-74d4-4b5f-99f5-8b426c906081 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:04.180573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ac0870-74d4-4b5f-99f5-8b426c906081 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:04.215406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99485af1-f3df-480d-9612-c09b54de4ef3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:04.216151 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99485af1-f3df-480d-9612-c09b54de4ef3 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:04.216428 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99485af1-f3df-480d-9612-c09b54de4ef3 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:04.216750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99485af1-f3df-480d-9612-c09b54de4ef3 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:04.217412 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99485af1-f3df-480d-9612-c09b54de4ef3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:04.217712 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99485af1-f3df-480d-9612-c09b54de4ef3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:04.219701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99485af1-f3df-480d-9612-c09b54de4ef3 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:04.228543 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 397/786] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:28:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:04.233864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa6f70c-9493-4f0a-8228-413bd7f06664 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:04.252376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99485af1-f3df-480d-9612-c09b54de4ef3 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:04.278940 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:04.373827 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99485af1-f3df-480d-9612-c09b54de4ef3 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:04.411219 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1898188-4f10-4bfc-9a13-643fcff0b1ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:04.412238 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1898188-4f10-4bfc-9a13-643fcff0b1ab None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:04.412722 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1898188-4f10-4bfc-9a13-643fcff0b1ab None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:04.413197 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1898188-4f10-4bfc-9a13-643fcff0b1ab None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:04.424367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1898188-4f10-4bfc-9a13-643fcff0b1ab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:28:04.430275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-a1898188-4f10-4bfc-9a13-643fcff0b1ab None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:28:04.430275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:04.430275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:04.430275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:04.430275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:04.430275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:04.430275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:04.430275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:04.430275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:04.430275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:04.430275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:04.430275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:04.430275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:28:04.430275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:28:04.430275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:28:04.430275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:28:04.431262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:28:04.431262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:28:04.431262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:28:04.431262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:28:04.431262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:04.431262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:04.431262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:04.431262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:04.431262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:04.431262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:04.431262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:04.431262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:04.431262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:04.431262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:04.431262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:04.431262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:04.432129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:04.432129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:04.432129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:04.432129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:04.432129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:28:04.432129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:28:04.432129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:28:04.432129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:28:04.432129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:28:04.432129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:28:04.432129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:28:04.432129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:04.434748 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 398/787] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:28:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:04.444679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1898188-4f10-4bfc-9a13-643fcff0b1ab None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:04.473534 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa6f70c-9493-4f0a-8228-413bd7f06664 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:04.510948 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e3208d-08d0-4509-91d3-a3e41a438dc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:04.511509 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e3208d-08d0-4509-91d3-a3e41a438dc9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:04.511785 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e3208d-08d0-4509-91d3-a3e41a438dc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:04.512055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e3208d-08d0-4509-91d3-a3e41a438dc9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:04.538006 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e3208d-08d0-4509-91d3-a3e41a438dc9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:04.539490 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e3208d-08d0-4509-91d3-a3e41a438dc9 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:04.567333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-55e3208d-08d0-4509-91d3-a3e41a438dc9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:04.567333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:04.567333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:04.567333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:04.567333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:04.567333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:04.567333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:04.567333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:04.567333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:04.567333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:04.567333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:04.567333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:04.567333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:04.567333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:04.567333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:04.567333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:04.568368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:04.568368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:04.568368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:04.568368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:04.568368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:04.568368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:04.568368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:04.568368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:04.568368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:04.568368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:04.568368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:04.568368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:04.568368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:04.568368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:04.568368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:04.568368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:04.568368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:04.569259 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:04.569259 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:04.569259 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:04.571349 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 390/788] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 338 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:04.580162 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e3208d-08d0-4509-91d3-a3e41a438dc9 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:04.625192 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1898188-4f10-4bfc-9a13-643fcff0b1ab None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:04.627840 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:04.632956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:04.692183 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e3208d-08d0-4509-91d3-a3e41a438dc9 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:04.705980 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16c1383-1f1c-4047-9967-f3e217d91bca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:04.706651 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16c1383-1f1c-4047-9967-f3e217d91bca None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:04.706885 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16c1383-1f1c-4047-9967-f3e217d91bca None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:04.707188 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16c1383-1f1c-4047-9967-f3e217d91bca None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:04.707915 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16c1383-1f1c-4047-9967-f3e217d91bca None admin] RBAC: Adding query filter params (name=admin) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:04.708231 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16c1383-1f1c-4047-9967-f3e217d91bca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:04.709894 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16c1383-1f1c-4047-9967-f3e217d91bca None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:04.719940 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 399/789] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:28:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:04.726279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16c1383-1f1c-4047-9967-f3e217d91bca None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:04.743909 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425469d9-31d1-4573-a05c-9fbb11b8afb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:04.744617 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425469d9-31d1-4573-a05c-9fbb11b8afb6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:04.744946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425469d9-31d1-4573-a05c-9fbb11b8afb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:04.745232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425469d9-31d1-4573-a05c-9fbb11b8afb6 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:04.745866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-425469d9-31d1-4573-a05c-9fbb11b8afb6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:04.746166 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-425469d9-31d1-4573-a05c-9fbb11b8afb6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:04.747541 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-425469d9-31d1-4573-a05c-9fbb11b8afb6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:04.766471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 391/790] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:04.766471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2b0e12-537e-4ba4-ac4c-f0856d1a7267 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:04.766471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2b0e12-537e-4ba4-ac4c-f0856d1a7267 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:04.766471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2b0e12-537e-4ba4-ac4c-f0856d1a7267 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:04.766471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 392/791] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:04.766471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2b0e12-537e-4ba4-ac4c-f0856d1a7267 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:04.870372 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16c1383-1f1c-4047-9967-f3e217d91bca None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:04.877551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2b0e12-537e-4ba4-ac4c-f0856d1a7267 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:04.886218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:04.918208 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2611d1b8-c8f9-4921-a4ea-2f4869156c6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:04.919156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2611d1b8-c8f9-4921-a4ea-2f4869156c6c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:04.919591 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2611d1b8-c8f9-4921-a4ea-2f4869156c6c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:04.919753 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87b2533-d9ab-4fbf-a68b-89ca1490922d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:04.920245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2611d1b8-c8f9-4921-a4ea-2f4869156c6c None admin] PATH_INFO: `/v3/users/swift` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:04.920551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87b2533-d9ab-4fbf-a68b-89ca1490922d None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:04.920934 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87b2533-d9ab-4fbf-a68b-89ca1490922d None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:04.921303 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87b2533-d9ab-4fbf-a68b-89ca1490922d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:04.933276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a87b2533-d9ab-4fbf-a68b-89ca1490922d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:04.933534 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2611d1b8-c8f9-4921-a4ea-2f4869156c6c None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:04.935311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2611d1b8-c8f9-4921-a4ea-2f4869156c6c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:04.935454 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a87b2533-d9ab-4fbf-a68b-89ca1490922d None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:04.940230 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:04.954947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-2611d1b8-c8f9-4921-a4ea-2f4869156c6c None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 16 03:28:04.954947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:04.954947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:04.954947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:04.954947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:04.954947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:04.954947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:04.954947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:04.954947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:04.954947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:04.954947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:04.954947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:04.954947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:28:04.954947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:28:04.954947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:28:04.954947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:28:04.956140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:04.956140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:04.956140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:28:04.956140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:04.956140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:28:04.956140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:04.956140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:04.956140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:04.956140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:04.956140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:04.956140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:04.956140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:04.956140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:04.956140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:04.956140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:04.956140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:04.956140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:04.956140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:04.957328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:04.957328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:04.957328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:28:04.957328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:28:04.957328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:28:04.957328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:28:04.957328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:28:04.957328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:28:04.957328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 16 03:28:04.957328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:04.958309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-a87b2533-d9ab-4fbf-a68b-89ca1490922d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:04.958309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:04.958309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:04.958309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:04.958309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:04.958309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:04.958309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:04.958309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:04.958309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:04.958309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:04.958309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:04.958309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:04.958309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:04.958309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:04.958309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:04.958309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:04.959462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:04.959462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:04.959462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:04.959462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:04.959462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:04.959462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:04.959462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:04.959462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:04.959462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:04.959462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:04.959462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:04.959462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:04.959462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:04.959462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:04.959462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:04.959462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:04.959462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:04.960596 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:04.960596 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:04.960596 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:04.966412 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 393/793] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:28:04 2022] GET /identity/v3/users/swift => generated 83 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:04.966766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 400/793] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:04.967783 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87b2533-d9ab-4fbf-a68b-89ca1490922d None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:04.968359 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a305507-0700-4f40-80be-7ba7b2caa02a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:04.968893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a305507-0700-4f40-80be-7ba7b2caa02a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:04.969375 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a305507-0700-4f40-80be-7ba7b2caa02a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:04.990210 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-3a305507-0700-4f40-80be-7ba7b2caa02a None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:04.991782 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3a305507-0700-4f40-80be-7ba7b2caa02a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:05.075227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87b2533-d9ab-4fbf-a68b-89ca1490922d None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:05.083340 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 394/794] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:04 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:05.084468 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a305507-0700-4f40-80be-7ba7b2caa02a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:05.154939 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc6307b-f6e2-4afd-9421-b428726533bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:05.156839 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc6307b-f6e2-4afd-9421-b428726533bc None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:05.157437 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc6307b-f6e2-4afd-9421-b428726533bc None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:05.157945 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc6307b-f6e2-4afd-9421-b428726533bc None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:05.159241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc6307b-f6e2-4afd-9421-b428726533bc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:05.159810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc6307b-f6e2-4afd-9421-b428726533bc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:05.162471 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc6307b-f6e2-4afd-9421-b428726533bc None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:05.190877 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:05.192233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 401/795] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:05.197675 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc6307b-f6e2-4afd-9421-b428726533bc None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:05.198811 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a305507-0700-4f40-80be-7ba7b2caa02a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:05.248936 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74deee7d-fd87-48a2-b80d-b6f621a0b0b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:05.249923 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74deee7d-fd87-48a2-b80d-b6f621a0b0b6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:05.250536 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74deee7d-fd87-48a2-b80d-b6f621a0b0b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:05.251317 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74deee7d-fd87-48a2-b80d-b6f621a0b0b6 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:05.252273 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74deee7d-fd87-48a2-b80d-b6f621a0b0b6 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:05.252803 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74deee7d-fd87-48a2-b80d-b6f621a0b0b6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:05.254446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74deee7d-fd87-48a2-b80d-b6f621a0b0b6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:05.273043 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 395/796] 10.222.0.49 () {56 vars in 1135 bytes} [Fri Sep 16 03:28:05 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:05.274783 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:05.279191 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74deee7d-fd87-48a2-b80d-b6f621a0b0b6 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:05.324084 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc6307b-f6e2-4afd-9421-b428726533bc None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:05.364139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9865dc1f-5079-4039-bd20-569e32fda010 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:05.365024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9865dc1f-5079-4039-bd20-569e32fda010 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:05.365539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9865dc1f-5079-4039-bd20-569e32fda010 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:05.365972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9865dc1f-5079-4039-bd20-569e32fda010 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:05.373675 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9865dc1f-5079-4039-bd20-569e32fda010 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:05.375350 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9865dc1f-5079-4039-bd20-569e32fda010 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:05.388277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74deee7d-fd87-48a2-b80d-b6f621a0b0b6 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:05.390255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-9865dc1f-5079-4039-bd20-569e32fda010 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 16 03:28:05.390255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:05.390255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:05.390255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:05.390255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:05.390255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:05.390255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:05.390255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:05.390255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:05.390255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:05.390255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:05.390255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:05.390255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:28:05.390255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:28:05.390255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:28:05.390255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:28:05.392178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:05.392178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:05.392178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:28:05.392178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:05.392178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:28:05.392178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:05.392178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:05.392178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:05.392178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:05.392178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:05.392178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:05.392178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:05.392178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:05.392178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:05.392178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:05.392178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:05.392178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:05.392178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:05.393249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:05.393249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:05.393249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:28:05.393249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:28:05.393249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:28:05.393249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:28:05.393249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:28:05.393249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:28:05.393249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 16 03:28:05.393249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:05.395568 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 402/797] 10.222.0.49 () {56 vars in 1093 bytes} [Fri Sep 16 03:28:05 2022] GET /identity/v3/users/nova => generated 82 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:05.401433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0012ef-02e4-4567-8d5d-79f6184f2c29 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:05.401960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0012ef-02e4-4567-8d5d-79f6184f2c29 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:05.402491 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0012ef-02e4-4567-8d5d-79f6184f2c29 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:05.404255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 403/798] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:05 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:05.405391 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0012ef-02e4-4567-8d5d-79f6184f2c29 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:05.428428 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04a78ff-353f-4b90-99ff-9dd2e555ee3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:05.429248 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04a78ff-353f-4b90-99ff-9dd2e555ee3c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:05.429509 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04a78ff-353f-4b90-99ff-9dd2e555ee3c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:05.429858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04a78ff-353f-4b90-99ff-9dd2e555ee3c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:05.440479 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04a78ff-353f-4b90-99ff-9dd2e555ee3c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:05.442056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04a78ff-353f-4b90-99ff-9dd2e555ee3c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:05.458091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-f04a78ff-353f-4b90-99ff-9dd2e555ee3c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:05.458091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:05.458091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:05.458091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:05.458091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:05.458091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:05.458091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:05.458091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:05.458091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:05.458091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:05.458091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:05.458091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:05.458091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:05.458091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:05.458091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:05.458091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:05.459285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:05.459285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:05.459285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:05.459285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:05.459285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:05.459285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:05.459285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:05.459285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:05.459285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:05.459285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:05.459285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:05.459285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:05.459285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:05.459285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:05.459285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:05.459285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:05.459285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:05.460218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:05.460218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:05.460218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:05.463870 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 396/799] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:05.465230 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a78738-06b4-47d5-98ff-5a3405dc681e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:05.465441 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a78738-06b4-47d5-98ff-5a3405dc681e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:05.465723 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a78738-06b4-47d5-98ff-5a3405dc681e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:05.478665 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:05.485685 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f6a78738-06b4-47d5-98ff-5a3405dc681e None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:05.487080 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f6a78738-06b4-47d5-98ff-5a3405dc681e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:05.528847 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0012ef-02e4-4567-8d5d-79f6184f2c29 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:05.548282 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:05.560306 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 397/800] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:05 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:05.561041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a78738-06b4-47d5-98ff-5a3405dc681e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:05.574484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aebceff-6326-4168-a212-f72417eb143b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:05.575413 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aebceff-6326-4168-a212-f72417eb143b None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:05.576112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aebceff-6326-4168-a212-f72417eb143b None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:05.576611 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aebceff-6326-4168-a212-f72417eb143b None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:05.578633 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aebceff-6326-4168-a212-f72417eb143b None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:05.579329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aebceff-6326-4168-a212-f72417eb143b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:05.580820 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aebceff-6326-4168-a212-f72417eb143b None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:05.605411 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 404/801] 10.222.0.49 () {56 vars in 1133 bytes} [Fri Sep 16 03:28:05 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:05.605411 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aebceff-6326-4168-a212-f72417eb143b None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:05.680449 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a78738-06b4-47d5-98ff-5a3405dc681e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:05.717408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3eaf75b-5034-47e2-8ce8-342a0f45f052 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:05.718172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3eaf75b-5034-47e2-8ce8-342a0f45f052 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:05.718558 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3eaf75b-5034-47e2-8ce8-342a0f45f052 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:05.719595 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3eaf75b-5034-47e2-8ce8-342a0f45f052 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:05.720641 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3eaf75b-5034-47e2-8ce8-342a0f45f052 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:05.721446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3eaf75b-5034-47e2-8ce8-342a0f45f052 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:05.722662 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3eaf75b-5034-47e2-8ce8-342a0f45f052 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:05.725435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:05.741569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 398/802] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:05.742596 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3eaf75b-5034-47e2-8ce8-342a0f45f052 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:05.745569 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aebceff-6326-4168-a212-f72417eb143b None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:05.792892 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65537ff-cd7a-4a43-913e-82f31b9533c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:05.793528 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65537ff-cd7a-4a43-913e-82f31b9533c4 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:05.793809 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65537ff-cd7a-4a43-913e-82f31b9533c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:05.794084 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65537ff-cd7a-4a43-913e-82f31b9533c4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:05.811289 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65537ff-cd7a-4a43-913e-82f31b9533c4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:28:05.818555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-a65537ff-cd7a-4a43-913e-82f31b9533c4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:28:05.818555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:05.818555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:05.818555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:05.818555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:05.818555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:05.818555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:05.818555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:05.818555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:05.818555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:05.818555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:05.818555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:05.818555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:28:05.818555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:28:05.818555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:28:05.818555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:28:05.819539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:28:05.819539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:28:05.819539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:28:05.819539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:28:05.819539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:05.819539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:05.819539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:05.819539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:05.819539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:05.819539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:05.819539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:05.819539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:05.819539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:05.819539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:05.819539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:05.819539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:05.820389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:05.820389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:05.820389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:05.820389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:05.820389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:28:05.820389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:28:05.820389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:28:05.820389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:28:05.820389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:28:05.820389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:28:05.820389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:28:05.820389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:05.823375 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 405/803] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:28:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:05.824011 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65537ff-cd7a-4a43-913e-82f31b9533c4 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:05.857524 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3eaf75b-5034-47e2-8ce8-342a0f45f052 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:05.870643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:05.897058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bbad36-1572-425b-bea1-925ee019bbb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:05.897683 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bbad36-1572-425b-bea1-925ee019bbb7 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:05.897965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bbad36-1572-425b-bea1-925ee019bbb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:05.898233 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bbad36-1572-425b-bea1-925ee019bbb7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:05.944549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65537ff-cd7a-4a43-913e-82f31b9533c4 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:05.967351 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:05.974697 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bbad36-1572-425b-bea1-925ee019bbb7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:05.976323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bbad36-1572-425b-bea1-925ee019bbb7 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:05.980897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcac18d-6e47-4891-8dfb-41d8bfa9bad9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:05.981744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcac18d-6e47-4891-8dfb-41d8bfa9bad9 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:05.982027 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcac18d-6e47-4891-8dfb-41d8bfa9bad9 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:05.982301 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcac18d-6e47-4891-8dfb-41d8bfa9bad9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:05.989994 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcac18d-6e47-4891-8dfb-41d8bfa9bad9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:05.991743 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcac18d-6e47-4891-8dfb-41d8bfa9bad9 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:06.003650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-0bcac18d-6e47-4891-8dfb-41d8bfa9bad9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:06.003650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:06.003650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:06.003650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:06.003650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:06.003650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:06.003650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:06.003650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:06.003650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:06.003650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:06.003650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:06.003650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:06.003650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:28:06.003650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:28:06.003650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:28:06.004698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:28:06.004698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:06.004698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:06.004698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:06.004698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:06.004698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:06.004698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:06.004698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:06.004698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:06.004698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:06.004698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:06.004698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:06.004698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:06.004698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:06.004698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:06.004698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:06.004698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:06.005602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:28:06.005602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:28:06.005602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:28:06.005602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:28:06.005602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:28:06.005602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:28:06.005602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:06.005602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:06.005602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 406/804] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:28:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:06.005602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcac18d-6e47-4891-8dfb-41d8bfa9bad9 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:06.081716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 399/805] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:28:05 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 03:28:06.082556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bbad36-1572-425b-bea1-925ee019bbb7 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:06.098877 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:06.104872 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcac18d-6e47-4891-8dfb-41d8bfa9bad9 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:06.138121 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3f7418-afbe-407e-ab44-a1d7cb885a9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:06.138645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3f7418-afbe-407e-ab44-a1d7cb885a9d None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:06.138952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3f7418-afbe-407e-ab44-a1d7cb885a9d None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:06.139223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3f7418-afbe-407e-ab44-a1d7cb885a9d None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:06.139872 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3f7418-afbe-407e-ab44-a1d7cb885a9d None admin] RBAC: Adding query filter params (name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:06.140171 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3f7418-afbe-407e-ab44-a1d7cb885a9d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:06.142244 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3f7418-afbe-407e-ab44-a1d7cb885a9d None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:06.151061 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 407/806] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:28:06 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:06.152334 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d199fa08-8696-46b8-9493-e2b130591d3a None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:06.152796 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d199fa08-8696-46b8-9493-e2b130591d3a None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:06.153236 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d199fa08-8696-46b8-9493-e2b130591d3a None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:06.158947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 408/807] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:06.158947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d199fa08-8696-46b8-9493-e2b130591d3a None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:06.190423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:06.196241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bbad36-1572-425b-bea1-925ee019bbb7 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:06.231135 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e022b3-b5cd-4790-9a30-be6b6d0a4346 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:06.231758 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e022b3-b5cd-4790-9a30-be6b6d0a4346 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:06.232035 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e022b3-b5cd-4790-9a30-be6b6d0a4346 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:06.232305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e022b3-b5cd-4790-9a30-be6b6d0a4346 None admin] PATH_INFO: `/v3/services/network` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:06.232874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e022b3-b5cd-4790-9a30-be6b6d0a4346 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:06.234221 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e022b3-b5cd-4790-9a30-be6b6d0a4346 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:06.242805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-72e022b3-b5cd-4790-9a30-be6b6d0a4346 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 16 03:28:06.242805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:06.242805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:06.242805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:06.242805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:06.242805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:06.242805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:06.242805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:06.242805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 16 03:28:06.242805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:06.242805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:06.242805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:06.242805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:06.242805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:06.242805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:06.242805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:06.242805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:06.243893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:06.243893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:06.243893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:06.243893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:06.243893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:06.243893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:06.243893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:06.243893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:06.243893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:06.243893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:06.243893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:06.243893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:06.243893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:06.243893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:06.243893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:06.243893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:06.244770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:06.244770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:06.244770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:06.244770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:06.244770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:06.244770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:06.244770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:06.244770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:06.244770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:06.244770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:06.244770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:06.244770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:06.244770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 16 03:28:06.244770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:06.244770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 400/808] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:28:06 2022] GET /identity/v3/services/network => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:06.245577 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e022b3-b5cd-4790-9a30-be6b6d0a4346 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:06.259089 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d199fa08-8696-46b8-9493-e2b130591d3a None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:06.293313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb802569-d245-4a9d-82dc-43e13d90254d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:06.293917 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb802569-d245-4a9d-82dc-43e13d90254d None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:06.294198 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb802569-d245-4a9d-82dc-43e13d90254d None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:06.294473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb802569-d245-4a9d-82dc-43e13d90254d None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:06.295194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb802569-d245-4a9d-82dc-43e13d90254d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:06.295500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb802569-d245-4a9d-82dc-43e13d90254d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:06.296974 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb802569-d245-4a9d-82dc-43e13d90254d None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:06.316663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 409/809] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:28:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:06.317431 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb802569-d245-4a9d-82dc-43e13d90254d None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:06.335249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:06.347618 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e022b3-b5cd-4790-9a30-be6b6d0a4346 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:06.380400 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83688e93-ab92-41f9-b575-a2cf193710ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:06.381053 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83688e93-ab92-41f9-b575-a2cf193710ee None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:06.381334 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83688e93-ab92-41f9-b575-a2cf193710ee None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:06.381610 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83688e93-ab92-41f9-b575-a2cf193710ee None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:06.392744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83688e93-ab92-41f9-b575-a2cf193710ee None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:06.394119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83688e93-ab92-41f9-b575-a2cf193710ee None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:06.408768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-83688e93-ab92-41f9-b575-a2cf193710ee None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:06.408768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:06.408768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:06.408768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:06.408768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:06.408768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:06.408768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:06.408768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:06.408768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:06.408768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:06.408768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:06.408768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:06.408768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:06.408768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:06.408768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:06.408768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:06.409890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:06.409890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:06.409890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:06.409890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:06.409890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:06.409890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:06.409890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:06.409890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:06.409890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:06.409890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:06.409890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:06.409890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:06.409890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:06.409890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:06.409890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:06.409890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:06.409890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:06.410779 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:06.410779 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:06.410779 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:06.410779 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 401/810] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:06.411054 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182ccedf-4c1d-48b4-ad88-b83035923aa0 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:06.411262 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182ccedf-4c1d-48b4-ad88-b83035923aa0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:06.411539 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182ccedf-4c1d-48b4-ad88-b83035923aa0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:06.420750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb802569-d245-4a9d-82dc-43e13d90254d None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:06.428857 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-182ccedf-4c1d-48b4-ad88-b83035923aa0 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:06.430048 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-182ccedf-4c1d-48b4-ad88-b83035923aa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:06.437256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:06.454144 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678df421-2578-4125-8c53-0e2fa2f6053f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:06.454772 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678df421-2578-4125-8c53-0e2fa2f6053f None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:06.455169 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678df421-2578-4125-8c53-0e2fa2f6053f None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:06.455447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678df421-2578-4125-8c53-0e2fa2f6053f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:06.465497 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678df421-2578-4125-8c53-0e2fa2f6053f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:06.466998 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678df421-2578-4125-8c53-0e2fa2f6053f None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:06.481946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-678df421-2578-4125-8c53-0e2fa2f6053f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:06.481946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:06.481946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:06.481946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:06.481946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:06.481946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:06.481946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:06.481946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:06.481946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:06.481946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:06.481946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:06.481946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:06.481946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:06.481946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:06.481946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:06.481946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:06.483024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:06.483024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:06.483024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:06.483024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:06.483024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:06.483024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:06.483024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:06.483024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:06.483024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:06.483024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:06.483024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:06.483024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:06.483024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:06.483024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:06.483024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:06.483024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:06.483024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:06.483923 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:06.483923 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:06.483923 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:06.483923 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 410/811] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:06.484340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678df421-2578-4125-8c53-0e2fa2f6053f None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:06.501759 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 402/812] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:06 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:06.502662 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182ccedf-4c1d-48b4-ad88-b83035923aa0 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:06.568293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:06.601423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678df421-2578-4125-8c53-0e2fa2f6053f None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:06.624686 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182ccedf-4c1d-48b4-ad88-b83035923aa0 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:06.637041 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee0f072-ace1-47f8-87f4-fed3eba7880a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:06.637654 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee0f072-ace1-47f8-87f4-fed3eba7880a None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:06.638046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee0f072-ace1-47f8-87f4-fed3eba7880a None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:06.638328 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee0f072-ace1-47f8-87f4-fed3eba7880a None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:06.639096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee0f072-ace1-47f8-87f4-fed3eba7880a None admin] RBAC: Adding query filter params (name=network) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:06.639406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee0f072-ace1-47f8-87f4-fed3eba7880a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:06.640852 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee0f072-ace1-47f8-87f4-fed3eba7880a None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:06.650325 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 411/813] 10.222.0.49 () {56 vars in 1112 bytes} [Fri Sep 16 03:28:06 2022] GET /identity/v3/services?name=network => generated 124 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:06.651095 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee0f072-ace1-47f8-87f4-fed3eba7880a None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:06.659939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd259baa-fde8-4850-9920-4ff1bd7f42fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:06.660609 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd259baa-fde8-4850-9920-4ff1bd7f42fb None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:06.660890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd259baa-fde8-4850-9920-4ff1bd7f42fb None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:06.661170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd259baa-fde8-4850-9920-4ff1bd7f42fb None admin] PATH_INFO: `/v3/projects/0bb7350474f349d6aee197ff35536167/users/aa30da9972e140d7905c522576681429/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:06.663142 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd259baa-fde8-4850-9920-4ff1bd7f42fb None admin] RBAC: Authorizing `identity:create_grant(project_id=0bb7350474f349d6aee197ff35536167, user_id=aa30da9972e140d7905c522576681429, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:06.664552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd259baa-fde8-4850-9920-4ff1bd7f42fb None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:06.742788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 403/814] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:28:06 2022] PUT /identity/v3/projects/0bb7350474f349d6aee197ff35536167/users/aa30da9972e140d7905c522576681429/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:28:06.743841 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd259baa-fde8-4850-9920-4ff1bd7f42fb None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:06.754272 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:06.760962 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee0f072-ace1-47f8-87f4-fed3eba7880a None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:06.796024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:06.802074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98d73e9-2ebb-4763-a128-f08ca3e482c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:06.802711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98d73e9-2ebb-4763-a128-f08ca3e482c5 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:06.803028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98d73e9-2ebb-4763-a128-f08ca3e482c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:06.803310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98d73e9-2ebb-4763-a128-f08ca3e482c5 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:06.804235 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98d73e9-2ebb-4763-a128-f08ca3e482c5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:06.804422 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98d73e9-2ebb-4763-a128-f08ca3e482c5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:06.805812 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98d73e9-2ebb-4763-a128-f08ca3e482c5 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:06.819559 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 412/815] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:06.820552 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98d73e9-2ebb-4763-a128-f08ca3e482c5 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:06.871140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd259baa-fde8-4850-9920-4ff1bd7f42fb None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:06.910998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158489c6-c8a2-40db-a502-48651661bdc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:06.911536 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158489c6-c8a2-40db-a502-48651661bdc7 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:06.911811 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158489c6-c8a2-40db-a502-48651661bdc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:06.912082 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158489c6-c8a2-40db-a502-48651661bdc7 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:06.912735 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158489c6-c8a2-40db-a502-48651661bdc7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:06.913036 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158489c6-c8a2-40db-a502-48651661bdc7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:06.914437 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158489c6-c8a2-40db-a502-48651661bdc7 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:06.931690 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 404/816] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:06.931977 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158489c6-c8a2-40db-a502-48651661bdc7 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:06.954313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98d73e9-2ebb-4763-a128-f08ca3e482c5 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:06.996965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1af70c-80eb-46ed-8774-27c504f5ad06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:06.997576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1af70c-80eb-46ed-8774-27c504f5ad06 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:06.997853 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1af70c-80eb-46ed-8774-27c504f5ad06 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:06.998221 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1af70c-80eb-46ed-8774-27c504f5ad06 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:06.998852 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1af70c-80eb-46ed-8774-27c504f5ad06 None admin] RBAC: Adding query filter params (type=network) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:06.999180 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1af70c-80eb-46ed-8774-27c504f5ad06 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:07.000740 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1af70c-80eb-46ed-8774-27c504f5ad06 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:07.005756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 413/817] 10.222.0.49 () {56 vars in 1112 bytes} [Fri Sep 16 03:28:06 2022] GET /identity/v3/services?type=network => generated 350 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:07.006557 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1af70c-80eb-46ed-8774-27c504f5ad06 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:07.019240 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:07.061569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158489c6-c8a2-40db-a502-48651661bdc7 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:07.068120 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:07.098838 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d04828c-db3a-4e3a-bb73-d61e97ae8a93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:07.099507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d04828c-db3a-4e3a-bb73-d61e97ae8a93 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:07.099794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d04828c-db3a-4e3a-bb73-d61e97ae8a93 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:07.100071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d04828c-db3a-4e3a-bb73-d61e97ae8a93 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:07.101020 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d04828c-db3a-4e3a-bb73-d61e97ae8a93 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:07.102113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d04828c-db3a-4e3a-bb73-d61e97ae8a93 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:07.111374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-6d04828c-db3a-4e3a-bb73-d61e97ae8a93 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 16 03:28:07.111374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:07.111374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:07.111374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:07.111374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:07.111374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:07.111374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:07.111374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:07.111374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 16 03:28:07.111374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:07.111374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:07.111374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:07.111374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:07.111374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:07.111374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:07.111374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:07.111374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:07.112366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:07.112366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:07.112366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:07.112366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:07.112366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:07.112366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:07.112366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:07.112366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:07.112366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:07.112366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:07.112366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:07.112366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:07.112366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:07.112366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:07.112366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:07.112366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:07.116171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:07.116171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:07.116171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:07.116171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:07.116171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:07.116171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:07.116171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:07.116171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:07.116171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:07.116171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:07.116171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:07.116171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:07.116171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 16 03:28:07.116171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:07.116921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1af70c-80eb-46ed-8774-27c504f5ad06 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:07.117006 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 405/818] 10.222.0.49 () {56 vars in 1129 bytes} [Fri Sep 16 03:28:06 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:07.117006 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d04828c-db3a-4e3a-bb73-d61e97ae8a93 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:07.153503 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3820b5ff-b71c-4c8e-ad3d-aec0ee2495e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:07.154128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3820b5ff-b71c-4c8e-ad3d-aec0ee2495e9 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:07.154402 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3820b5ff-b71c-4c8e-ad3d-aec0ee2495e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:07.154669 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3820b5ff-b71c-4c8e-ad3d-aec0ee2495e9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:07.161890 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3820b5ff-b71c-4c8e-ad3d-aec0ee2495e9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:07.163296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3820b5ff-b71c-4c8e-ad3d-aec0ee2495e9 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:07.177353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-3820b5ff-b71c-4c8e-ad3d-aec0ee2495e9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:07.177353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:07.177353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:07.177353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:07.177353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:07.177353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:07.177353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:07.177353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:07.177353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:07.177353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:07.177353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:07.177353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:07.177353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:28:07.177353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:28:07.177353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:28:07.178332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:28:07.178332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:07.178332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:07.178332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:07.178332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:07.178332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:07.178332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:07.178332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:07.178332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:07.178332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:07.178332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:07.178332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:07.178332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:07.178332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:07.178332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:07.178332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:07.178332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:07.179752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:28:07.179752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:28:07.179752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:28:07.179752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:28:07.179752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:28:07.179752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:28:07.179752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:07.179752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:07.179752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 414/819] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:28:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:07.180481 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3820b5ff-b71c-4c8e-ad3d-aec0ee2495e9 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:07.238754 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d04828c-db3a-4e3a-bb73-d61e97ae8a93 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:07.268414 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:07.279230 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc84d7b3-2b3a-4ddc-baf3-940fadec0425 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:07.280139 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc84d7b3-2b3a-4ddc-baf3-940fadec0425 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:07.280678 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc84d7b3-2b3a-4ddc-baf3-940fadec0425 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:07.281061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc84d7b3-2b3a-4ddc-baf3-940fadec0425 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:07.285207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3820b5ff-b71c-4c8e-ad3d-aec0ee2495e9 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:07.290830 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc84d7b3-2b3a-4ddc-baf3-940fadec0425 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:07.292745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc84d7b3-2b3a-4ddc-baf3-940fadec0425 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:07.301495 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:07.307997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-fc84d7b3-2b3a-4ddc-baf3-940fadec0425 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 03:28:07.307997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:07.307997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:07.307997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:07.307997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:07.307997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:07.307997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:07.307997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:07.307997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:07.307997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:07.307997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:07.307997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:07.307997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:28:07.307997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:28:07.307997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:28:07.307997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:28:07.309235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:07.309235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:07.309235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:28:07.309235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:07.309235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:28:07.309235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:07.309235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:07.309235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:07.309235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:07.309235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:07.309235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:07.309235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:07.309235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:07.309235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:07.309235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:07.309235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:07.309235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:07.309235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:07.310423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:07.310423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:07.310423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:28:07.310423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:28:07.310423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:28:07.310423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:28:07.310423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:28:07.310423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:28:07.310423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 03:28:07.310423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:07.314567 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 406/820] 10.222.0.49 () {56 vars in 1117 bytes} [Fri Sep 16 03:28:07 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:07.316020 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc84d7b3-2b3a-4ddc-baf3-940fadec0425 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:07.332447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff2ee72-2715-4f3f-90b7-b948fe34c28d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:07.333069 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff2ee72-2715-4f3f-90b7-b948fe34c28d None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:07.333348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff2ee72-2715-4f3f-90b7-b948fe34c28d None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:07.333619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff2ee72-2715-4f3f-90b7-b948fe34c28d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:07.334290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff2ee72-2715-4f3f-90b7-b948fe34c28d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:07.335703 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff2ee72-2715-4f3f-90b7-b948fe34c28d None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:07.418103 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc84d7b3-2b3a-4ddc-baf3-940fadec0425 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:07.440537 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 415/821] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:28:07 2022] POST /identity/v3/endpoints => generated 337 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:28:07.441637 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff2ee72-2715-4f3f-90b7-b948fe34c28d None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:07.474562 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308781db-070a-4a0d-98da-8922eaac1b20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:07.475404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308781db-070a-4a0d-98da-8922eaac1b20 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:07.475795 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308781db-070a-4a0d-98da-8922eaac1b20 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:07.476145 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308781db-070a-4a0d-98da-8922eaac1b20 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:07.476871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308781db-070a-4a0d-98da-8922eaac1b20 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:07.477246 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308781db-070a-4a0d-98da-8922eaac1b20 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:07.478927 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308781db-070a-4a0d-98da-8922eaac1b20 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:07.488946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 407/822] 10.222.0.49 () {56 vars in 1124 bytes} [Fri Sep 16 03:28:07 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:07.488946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308781db-070a-4a0d-98da-8922eaac1b20 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:07.542639 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:07.603401 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff2ee72-2715-4f3f-90b7-b948fe34c28d None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:07.606714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308781db-070a-4a0d-98da-8922eaac1b20 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:07.641946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e436a860-26ba-49b9-899d-e4b257b6b02a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:07.642691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e436a860-26ba-49b9-899d-e4b257b6b02a None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:07.643083 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e436a860-26ba-49b9-899d-e4b257b6b02a None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:07.643431 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e436a860-26ba-49b9-899d-e4b257b6b02a None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:07.644341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e436a860-26ba-49b9-899d-e4b257b6b02a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:07.644822 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e436a860-26ba-49b9-899d-e4b257b6b02a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:07.646524 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e436a860-26ba-49b9-899d-e4b257b6b02a None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:07.648440 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1147d85d-fb3f-4169-b2ea-47366ce5efb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:07.649334 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1147d85d-fb3f-4169-b2ea-47366ce5efb9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:07.649835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1147d85d-fb3f-4169-b2ea-47366ce5efb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:07.650323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1147d85d-fb3f-4169-b2ea-47366ce5efb9 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:07.651386 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1147d85d-fb3f-4169-b2ea-47366ce5efb9 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:07.651905 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1147d85d-fb3f-4169-b2ea-47366ce5efb9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:07.653609 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1147d85d-fb3f-4169-b2ea-47366ce5efb9 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:07.662817 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 416/823] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:28:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:07.663783 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e436a860-26ba-49b9-899d-e4b257b6b02a None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:07.675294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 408/824] 10.222.0.49 () {56 vars in 1149 bytes} [Fri Sep 16 03:28:07 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:07.676432 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1147d85d-fb3f-4169-b2ea-47366ce5efb9 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:07.712005 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:07.794875 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1147d85d-fb3f-4169-b2ea-47366ce5efb9 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:07.798866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:07.801825 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e436a860-26ba-49b9-899d-e4b257b6b02a None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:07.831606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432851ea-78e9-452b-96b2-2b5c9ed8f6f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:07.833073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432851ea-78e9-452b-96b2-2b5c9ed8f6f7 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:07.833357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432851ea-78e9-452b-96b2-2b5c9ed8f6f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:07.833634 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432851ea-78e9-452b-96b2-2b5c9ed8f6f7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:07.834358 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432851ea-78e9-452b-96b2-2b5c9ed8f6f7 None admin] RBAC: Adding query filter params (role.id=89c4b3a971224d468d33d2dc411230a2, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=ded1bcd10e62499a9aaa0ece5aa55c80) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:07.834671 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432851ea-78e9-452b-96b2-2b5c9ed8f6f7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:07.836139 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432851ea-78e9-452b-96b2-2b5c9ed8f6f7 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:07.837750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26bd88bc-b6bd-4be5-851d-f1025945a8dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:07.838352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bd88bc-b6bd-4be5-851d-f1025945a8dd None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:07.838634 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bd88bc-b6bd-4be5-851d-f1025945a8dd None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:07.838900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bd88bc-b6bd-4be5-851d-f1025945a8dd None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:07.839555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26bd88bc-b6bd-4be5-851d-f1025945a8dd None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:07.839859 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26bd88bc-b6bd-4be5-851d-f1025945a8dd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:07.841203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26bd88bc-b6bd-4be5-851d-f1025945a8dd None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:07.842245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 409/825] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:28:07 2022] GET /identity/v3/role_assignments?user.id=ded1bcd10e62499a9aaa0ece5aa55c80&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=89c4b3a971224d468d33d2dc411230a2 => generated 608 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:07.843044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432851ea-78e9-452b-96b2-2b5c9ed8f6f7 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:07.857305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 417/826] 10.222.0.49 () {56 vars in 1124 bytes} [Fri Sep 16 03:28:07 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:07.864320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26bd88bc-b6bd-4be5-851d-f1025945a8dd None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:07.976761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:07.983838 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432851ea-78e9-452b-96b2-2b5c9ed8f6f7 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:07.984106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26bd88bc-b6bd-4be5-851d-f1025945a8dd None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:08.027225 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6bcd2b-74f6-4944-aaf2-f6f02aece2fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:08.028021 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6bcd2b-74f6-4944-aaf2-f6f02aece2fc None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:08.028311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6bcd2b-74f6-4944-aaf2-f6f02aece2fc None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:08.028649 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6bcd2b-74f6-4944-aaf2-f6f02aece2fc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:08.036156 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1928e3e3-fca6-4c52-ae3f-c865d7e44b52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:08.037065 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1928e3e3-fca6-4c52-ae3f-c865d7e44b52 None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:08.038132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1928e3e3-fca6-4c52-ae3f-c865d7e44b52 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:08.038586 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1928e3e3-fca6-4c52-ae3f-c865d7e44b52 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:08.040003 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1928e3e3-fca6-4c52-ae3f-c865d7e44b52 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:08.041617 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6bcd2b-74f6-4944-aaf2-f6f02aece2fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:08.042042 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1928e3e3-fca6-4c52-ae3f-c865d7e44b52 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:08.043226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6bcd2b-74f6-4944-aaf2-f6f02aece2fc None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:08.061422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-1f6bcd2b-74f6-4944-aaf2-f6f02aece2fc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:08.061422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:08.061422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:08.061422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:08.061422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:08.061422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:08.061422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:08.061422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:08.061422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:08.061422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:08.061422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:08.061422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:08.061422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:08.061422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:08.061422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:08.061422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:08.065896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:08.065896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:08.065896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:08.065896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:08.065896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:08.065896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:08.065896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:08.065896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:08.065896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:08.065896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:08.065896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:08.065896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:08.065896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:08.065896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:08.065896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:08.065896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:08.065896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:08.070032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:08.070032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:08.070032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:08.070032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 410/827] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:08.070032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6bcd2b-74f6-4944-aaf2-f6f02aece2fc None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:08.094475 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:08.123333 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 418/828] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:28:07 2022] POST /identity/v3/endpoints => generated 346 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:28:08.188706 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6bcd2b-74f6-4944-aaf2-f6f02aece2fc None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:08.233801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2284253d-047e-4b0d-b9aa-fa9b1276420c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:08.234406 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2284253d-047e-4b0d-b9aa-fa9b1276420c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:08.234782 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2284253d-047e-4b0d-b9aa-fa9b1276420c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:08.235090 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2284253d-047e-4b0d-b9aa-fa9b1276420c None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:08.235759 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2284253d-047e-4b0d-b9aa-fa9b1276420c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:08.236064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2284253d-047e-4b0d-b9aa-fa9b1276420c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:08.237454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2284253d-047e-4b0d-b9aa-fa9b1276420c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:08.251412 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 411/829] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:08.257669 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1928e3e3-fca6-4c52-ae3f-c865d7e44b52 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:08.378583 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1928e3e3-fca6-4c52-ae3f-c865d7e44b52 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:08.414993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c3b765-5dad-4069-ad22-02f4cb28e8ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:08.416030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c3b765-5dad-4069-ad22-02f4cb28e8ca None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:08.416865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c3b765-5dad-4069-ad22-02f4cb28e8ca None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:08.417612 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c3b765-5dad-4069-ad22-02f4cb28e8ca None admin] PATH_INFO: `/v3/projects/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:08.428812 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c3b765-5dad-4069-ad22-02f4cb28e8ca None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:08.430538 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c3b765-5dad-4069-ad22-02f4cb28e8ca None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:08.443241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-b2c3b765-5dad-4069-ad22-02f4cb28e8ca None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:08.443241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:08.443241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:08.443241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:08.443241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:08.443241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:08.443241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:08.443241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:08.443241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:08.443241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:08.443241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:08.443241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:08.443241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:28:08.443241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:28:08.443241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:28:08.444287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:28:08.444287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:08.444287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:08.444287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:08.444287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:08.444287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:08.444287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:08.444287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:08.444287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:08.444287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:08.444287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:08.444287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:08.444287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:08.444287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:08.444287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:08.444287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:08.444287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:08.445207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:28:08.445207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:28:08.445207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:28:08.445207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:28:08.445207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:28:08.445207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:28:08.445207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:08.445207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:08.447273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 419/830] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:28:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:08.453652 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2284253d-047e-4b0d-b9aa-fa9b1276420c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:08.535450 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:08.558879 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2284253d-047e-4b0d-b9aa-fa9b1276420c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:08.593893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec22d9c-5a74-4c9b-8dbc-5ea224e31cdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:08.594507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec22d9c-5a74-4c9b-8dbc-5ea224e31cdf None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:08.594831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec22d9c-5a74-4c9b-8dbc-5ea224e31cdf None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:08.595087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec22d9c-5a74-4c9b-8dbc-5ea224e31cdf None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:08.595787 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec22d9c-5a74-4c9b-8dbc-5ea224e31cdf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:08.596092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec22d9c-5a74-4c9b-8dbc-5ea224e31cdf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:08.597514 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec22d9c-5a74-4c9b-8dbc-5ea224e31cdf None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:08.610838 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 412/831] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:28:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:08.616699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c3b765-5dad-4069-ad22-02f4cb28e8ca None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:08.638381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:08.737003 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c3b765-5dad-4069-ad22-02f4cb28e8ca None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:08.772556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784f0673-5b50-44e4-95c8-3fa89d7ea6bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:08.773222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784f0673-5b50-44e4-95c8-3fa89d7ea6bd None admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:08.773505 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784f0673-5b50-44e4-95c8-3fa89d7ea6bd None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:08.773779 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784f0673-5b50-44e4-95c8-3fa89d7ea6bd None admin] PATH_INFO: `/v3/projects/0bb7350474f349d6aee197ff35536167/users/d40810f61ead4c3da414194b325da80a/roles/974d839067e9411aa4cb850cee33f6ae` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:08.790368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784f0673-5b50-44e4-95c8-3fa89d7ea6bd None admin] RBAC: Authorizing `identity:create_grant(project_id=0bb7350474f349d6aee197ff35536167, user_id=d40810f61ead4c3da414194b325da80a, role_id=974d839067e9411aa4cb850cee33f6ae)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:08.791781 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784f0673-5b50-44e4-95c8-3fa89d7ea6bd None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:08.873864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 420/832] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:28:08 2022] PUT /identity/v3/projects/0bb7350474f349d6aee197ff35536167/users/d40810f61ead4c3da414194b325da80a/roles/974d839067e9411aa4cb850cee33f6ae => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:28:08.976202 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0a3aed-9a39-4d2b-9d35-995d76ba909b None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:08.976726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0a3aed-9a39-4d2b-9d35-995d76ba909b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:08.976726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0a3aed-9a39-4d2b-9d35-995d76ba909b None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:08.978447 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 413/833] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:08.984154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3928c601-4b46-4c89-b293-eddaf77df73a None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:08.985054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3928c601-4b46-4c89-b293-eddaf77df73a None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:08.985323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3928c601-4b46-4c89-b293-eddaf77df73a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:09.005787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-3928c601-4b46-4c89-b293-eddaf77df73a None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:09.007225 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3928c601-4b46-4c89-b293-eddaf77df73a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:09.072140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:09.083684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 421/834] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:08 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:09.374499 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0a3aed-9a39-4d2b-9d35-995d76ba909b None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:09.479915 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0a3aed-9a39-4d2b-9d35-995d76ba909b None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:09.514883 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607c6e0c-a497-4fb0-a72e-da37d85642af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:09.515630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607c6e0c-a497-4fb0-a72e-da37d85642af None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:09.516123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607c6e0c-a497-4fb0-a72e-da37d85642af None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:09.516662 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607c6e0c-a497-4fb0-a72e-da37d85642af None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:09.527337 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607c6e0c-a497-4fb0-a72e-da37d85642af None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:28:09.532340 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:09.533547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-607c6e0c-a497-4fb0-a72e-da37d85642af None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:28:09.533547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:09.533547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:09.533547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:09.533547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:09.533547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:09.533547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:09.533547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:09.533547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:09.533547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:09.533547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:09.533547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:09.533547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:28:09.533547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:28:09.533547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:28:09.533547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:28:09.534551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:28:09.534551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:28:09.534551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:28:09.534551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:28:09.534551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:09.534551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:09.534551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:09.534551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:09.534551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:09.534551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:09.534551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:09.534551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:09.534551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:09.534551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:09.534551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:09.534551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:09.535515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:09.535515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:09.535515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:09.535515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:09.535515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:28:09.535515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:28:09.535515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:28:09.535515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:28:09.535515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:28:09.535515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:28:09.535515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:28:09.535515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:09.535515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 414/835] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:28:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:09.544235 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3928c601-4b46-4c89-b293-eddaf77df73a None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:09.649723 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3928c601-4b46-4c89-b293-eddaf77df73a None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:09.685407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a833e2f-80dc-4b85-8dbb-da7d68f9e039 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:09.686031 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a833e2f-80dc-4b85-8dbb-da7d68f9e039 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:09.686309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a833e2f-80dc-4b85-8dbb-da7d68f9e039 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:09.686581 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a833e2f-80dc-4b85-8dbb-da7d68f9e039 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:09.687278 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a833e2f-80dc-4b85-8dbb-da7d68f9e039 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:09.688146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a833e2f-80dc-4b85-8dbb-da7d68f9e039 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:09.689527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a833e2f-80dc-4b85-8dbb-da7d68f9e039 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:09.698551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 422/836] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:28:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:09.704050 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607c6e0c-a497-4fb0-a72e-da37d85642af None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:09.805676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607c6e0c-a497-4fb0-a72e-da37d85642af None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:09.842606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a6e4c8-e895-44de-9d7e-8fc0bc3134ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:09.843264 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a6e4c8-e895-44de-9d7e-8fc0bc3134ef None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:09.843539 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a6e4c8-e895-44de-9d7e-8fc0bc3134ef None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:09.843859 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a6e4c8-e895-44de-9d7e-8fc0bc3134ef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:09.854084 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a6e4c8-e895-44de-9d7e-8fc0bc3134ef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:09.855543 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a6e4c8-e895-44de-9d7e-8fc0bc3134ef None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:09.870624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-f4a6e4c8-e895-44de-9d7e-8fc0bc3134ef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:09.870624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:09.870624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:09.870624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:09.870624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:09.870624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:09.870624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:09.870624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:09.870624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:09.870624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:09.870624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:09.870624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:09.870624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:09.870624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:09.870624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:09.870624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:09.871646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:09.871646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:09.871646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:09.871646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:09.871646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:09.871646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:09.871646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:09.871646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:09.871646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:09.871646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:09.871646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:09.871646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:09.871646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:09.871646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:09.871646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:09.871646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:09.871646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:09.872570 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:09.872570 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:09.872570 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:09.872570 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 415/837] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:09.879314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a833e2f-80dc-4b85-8dbb-da7d68f9e039 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:09.938533 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:09.984141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a833e2f-80dc-4b85-8dbb-da7d68f9e039 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:10.019177 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfe165d-193e-4e44-99ed-b222f858dbb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:10.019840 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfe165d-193e-4e44-99ed-b222f858dbb5 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:10.020129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfe165d-193e-4e44-99ed-b222f858dbb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:10.020405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfe165d-193e-4e44-99ed-b222f858dbb5 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:10.021050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfe165d-193e-4e44-99ed-b222f858dbb5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:10.021355 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfe165d-193e-4e44-99ed-b222f858dbb5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:10.022730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfe165d-193e-4e44-99ed-b222f858dbb5 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:10.035513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 423/838] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:10.040873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a6e4c8-e895-44de-9d7e-8fc0bc3134ef None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:10.105315 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:10.160520 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a6e4c8-e895-44de-9d7e-8fc0bc3134ef None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:10.198514 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0029fa0-6fc2-4796-bb76-bd9e557b0a72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:10.199442 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0029fa0-6fc2-4796-bb76-bd9e557b0a72 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:10.199885 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0029fa0-6fc2-4796-bb76-bd9e557b0a72 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:10.200304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff2cc57-567f-4685-b252-e1e3b07a7a12 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:10.200486 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0029fa0-6fc2-4796-bb76-bd9e557b0a72 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:10.200598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff2cc57-567f-4685-b252-e1e3b07a7a12 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:10.200852 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff2cc57-567f-4685-b252-e1e3b07a7a12 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:10.202444 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 424/839] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:10.207483 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc1afea-50b9-49ba-a4ba-c2a34a23df2c None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:10.207766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc1afea-50b9-49ba-a4ba-c2a34a23df2c None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:10.208042 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc1afea-50b9-49ba-a4ba-c2a34a23df2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:10.210055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0029fa0-6fc2-4796-bb76-bd9e557b0a72 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:10.211873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0029fa0-6fc2-4796-bb76-bd9e557b0a72 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:10.224702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-f0029fa0-6fc2-4796-bb76-bd9e557b0a72 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 16 03:28:10.224702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:10.224702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:10.224702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:10.224702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:10.224702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:10.224702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:10.224702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:10.224702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:10.224702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:10.224702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:10.224702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:10.224702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:28:10.224702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:28:10.224702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:28:10.224702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:28:10.225935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:10.225935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:10.225935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:28:10.225935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:10.225935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:28:10.225935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:10.225935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:10.225935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:10.225935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:10.225935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:10.225935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:10.225935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:10.225935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:10.225935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:10.225935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:10.225935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:10.225935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:10.225935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:10.227186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:10.227186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:10.227186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:28:10.227186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:28:10.227186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:28:10.227186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:28:10.227186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:28:10.227186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:28:10.227186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 16 03:28:10.227186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:10.228281 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-bbc1afea-50b9-49ba-a4ba-c2a34a23df2c None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:10.229557 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-bbc1afea-50b9-49ba-a4ba-c2a34a23df2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:10.229732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 416/840] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:28:10 2022] GET /identity/v3/users/swift => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:10.236138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0029fa0-6fc2-4796-bb76-bd9e557b0a72 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:10.328753 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 425/841] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:10 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 03:28:10.361010 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0029fa0-6fc2-4796-bb76-bd9e557b0a72 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:10.370088 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:10.395993 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba89166-76ea-4ef5-982b-84d5112f8be7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:10.396656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba89166-76ea-4ef5-982b-84d5112f8be7 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:10.396936 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba89166-76ea-4ef5-982b-84d5112f8be7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:10.397209 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba89166-76ea-4ef5-982b-84d5112f8be7 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:10.397929 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba89166-76ea-4ef5-982b-84d5112f8be7 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:10.398231 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba89166-76ea-4ef5-982b-84d5112f8be7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:10.399672 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba89166-76ea-4ef5-982b-84d5112f8be7 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:10.418311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 417/842] 10.222.0.49 () {56 vars in 1135 bytes} [Fri Sep 16 03:28:10 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:10.424376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc1afea-50b9-49ba-a4ba-c2a34a23df2c None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:10.476658 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:10.534890 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc1afea-50b9-49ba-a4ba-c2a34a23df2c None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:10.570957 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7381002-187e-484a-b196-9f39ba787e81 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:10.571643 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7381002-187e-484a-b196-9f39ba787e81 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:10.572123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7381002-187e-484a-b196-9f39ba787e81 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:10.572228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86bb1a92-5b05-41b5-802e-aaa9a91cb0c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:10.572836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bb1a92-5b05-41b5-802e-aaa9a91cb0c8 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:10.573121 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bb1a92-5b05-41b5-802e-aaa9a91cb0c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:10.573401 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bb1a92-5b05-41b5-802e-aaa9a91cb0c8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:10.574763 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 418/843] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:10 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:10.582200 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8273d4-abce-4bec-971a-34b40c4962e4 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:10.582463 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8273d4-abce-4bec-971a-34b40c4962e4 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:10.582746 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8273d4-abce-4bec-971a-34b40c4962e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:10.601316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-cc8273d4-abce-4bec-971a-34b40c4962e4 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:10.602530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cc8273d4-abce-4bec-971a-34b40c4962e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:10.648967 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86bb1a92-5b05-41b5-802e-aaa9a91cb0c8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:10.650493 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86bb1a92-5b05-41b5-802e-aaa9a91cb0c8 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:10.683036 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 419/844] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:10 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:10.683036 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8273d4-abce-4bec-971a-34b40c4962e4 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:10.789427 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:10.794901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8273d4-abce-4bec-971a-34b40c4962e4 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:10.825753 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:10.830713 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc0440f-c5cb-457b-8451-86f0ef8572ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:10.831380 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc0440f-c5cb-457b-8451-86f0ef8572ca None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:10.831657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc0440f-c5cb-457b-8451-86f0ef8572ca None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:10.831925 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc0440f-c5cb-457b-8451-86f0ef8572ca None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:10.832640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc0440f-c5cb-457b-8451-86f0ef8572ca None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:10.833924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc0440f-c5cb-457b-8451-86f0ef8572ca None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:10.842846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-9dc0440f-c5cb-457b-8451-86f0ef8572ca None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 16 03:28:10.842846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:10.842846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:10.842846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:10.842846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:10.842846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:10.842846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:10.842846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:10.842846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 16 03:28:10.842846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:10.842846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:10.842846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:10.842846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:10.842846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:10.842846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:10.842846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:10.844064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:10.844064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:10.844064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:10.844064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:10.844064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:10.844064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:10.844064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:10.844064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:10.844064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:10.844064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:10.844064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:10.844064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:10.844064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:10.844064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:10.844064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:10.844064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:10.844064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:10.845236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:10.845236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:10.845236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:10.845236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:10.845236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:10.845236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:10.845236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:10.845236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:10.845236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:10.845236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:10.845236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:10.845236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:10.845236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 16 03:28:10.845236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:10.849057 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 420/845] 10.222.0.49 () {56 vars in 1132 bytes} [Fri Sep 16 03:28:10 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:10.849952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc0440f-c5cb-457b-8451-86f0ef8572ca None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:10.897443 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 426/846] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:28:10 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 469 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 03:28:10.899516 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86bb1a92-5b05-41b5-802e-aaa9a91cb0c8 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:10.950393 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc0440f-c5cb-457b-8451-86f0ef8572ca None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:10.991541 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36271b99-dd22-497f-8fd3-2163ef98daf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:10.992195 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36271b99-dd22-497f-8fd3-2163ef98daf9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:10.992475 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36271b99-dd22-497f-8fd3-2163ef98daf9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:10.992751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36271b99-dd22-497f-8fd3-2163ef98daf9 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:10.993372 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36271b99-dd22-497f-8fd3-2163ef98daf9 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:10.993671 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36271b99-dd22-497f-8fd3-2163ef98daf9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:10.995134 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36271b99-dd22-497f-8fd3-2163ef98daf9 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:11.006527 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 421/847] 10.222.0.49 () {56 vars in 1126 bytes} [Fri Sep 16 03:28:10 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:11.007448 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36271b99-dd22-497f-8fd3-2163ef98daf9 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:11.025836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86bb1a92-5b05-41b5-802e-aaa9a91cb0c8 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:11.035421 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:11.067992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac48e7c-c95f-47ce-b071-cb2b531bc234 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:11.068538 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac48e7c-c95f-47ce-b071-cb2b531bc234 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:11.068818 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac48e7c-c95f-47ce-b071-cb2b531bc234 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:11.069091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac48e7c-c95f-47ce-b071-cb2b531bc234 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:11.080509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac48e7c-c95f-47ce-b071-cb2b531bc234 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:11.081946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac48e7c-c95f-47ce-b071-cb2b531bc234 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:11.085006 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:11.098000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-aac48e7c-c95f-47ce-b071-cb2b531bc234 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:11.098000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:11.098000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:11.098000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:11.098000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:11.098000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:11.098000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:11.098000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:11.098000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:11.098000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:11.098000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:11.098000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:11.098000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:11.098000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:11.098000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:11.098000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:11.099002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:11.099002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:11.099002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:11.099002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:11.099002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:11.099002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:11.099002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:11.099002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:11.099002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:11.099002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:11.099002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:11.099002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:11.099002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:11.099002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:11.099002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:11.099002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:11.099002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:11.099919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:11.099919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:11.099919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:11.099919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 427/848] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:11.099919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac48e7c-c95f-47ce-b071-cb2b531bc234 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:11.122758 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36271b99-dd22-497f-8fd3-2163ef98daf9 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:11.160790 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6db9f7-52ac-4432-922e-e03614c89d75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:11.161413 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6db9f7-52ac-4432-922e-e03614c89d75 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:11.161692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6db9f7-52ac-4432-922e-e03614c89d75 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:11.161965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6db9f7-52ac-4432-922e-e03614c89d75 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:11.162533 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6db9f7-52ac-4432-922e-e03614c89d75 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:11.163908 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6db9f7-52ac-4432-922e-e03614c89d75 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:11.172951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-3e6db9f7-52ac-4432-922e-e03614c89d75 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 16 03:28:11.172951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:11.172951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:11.172951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:11.172951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:11.172951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:11.172951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:11.172951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:11.172951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 16 03:28:11.172951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:11.172951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:11.172951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:11.172951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:11.172951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:11.172951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:11.172951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:11.172951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:11.174136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:11.174136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:11.174136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:11.174136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:11.174136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:11.174136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:11.174136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:11.174136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:11.174136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:11.174136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:11.174136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:11.174136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:11.174136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:11.174136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:11.174136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:11.174136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:11.175029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:11.175029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:11.175029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:11.175029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:11.175029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:11.175029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:11.175029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:11.175029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:11.175029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:11.175029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:11.175029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:11.175029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:11.175029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 16 03:28:11.175029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:11.175029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 422/849] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:28:11 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:11.175833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6db9f7-52ac-4432-922e-e03614c89d75 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:11.200781 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac48e7c-c95f-47ce-b071-cb2b531bc234 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:11.237613 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e7c86e-17ea-4dde-9b5c-3dfec9049439 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:11.238222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e7c86e-17ea-4dde-9b5c-3dfec9049439 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:11.238502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e7c86e-17ea-4dde-9b5c-3dfec9049439 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:11.238773 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e7c86e-17ea-4dde-9b5c-3dfec9049439 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:11.239426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e7c86e-17ea-4dde-9b5c-3dfec9049439 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:11.239726 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e7c86e-17ea-4dde-9b5c-3dfec9049439 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:11.241097 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e7c86e-17ea-4dde-9b5c-3dfec9049439 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:11.245387 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 428/850] 10.222.0.49 () {56 vars in 1126 bytes} [Fri Sep 16 03:28:11 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:11.246537 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef862d2-cc16-4d69-8dd8-6a288c1b1f97 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:11.246821 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef862d2-cc16-4d69-8dd8-6a288c1b1f97 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:11.247131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef862d2-cc16-4d69-8dd8-6a288c1b1f97 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:11.248321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 429/851] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:11 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:11.249042 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef862d2-cc16-4d69-8dd8-6a288c1b1f97 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:11.275235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:11.289373 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6db9f7-52ac-4432-922e-e03614c89d75 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:11.321054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:11.336602 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948e1e7c-6bd2-451f-a2b0-9992b7dd804a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:11.337232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948e1e7c-6bd2-451f-a2b0-9992b7dd804a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:11.337510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948e1e7c-6bd2-451f-a2b0-9992b7dd804a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:11.337897 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948e1e7c-6bd2-451f-a2b0-9992b7dd804a None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:11.338628 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948e1e7c-6bd2-451f-a2b0-9992b7dd804a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:11.339021 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948e1e7c-6bd2-451f-a2b0-9992b7dd804a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:11.340413 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948e1e7c-6bd2-451f-a2b0-9992b7dd804a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:11.354157 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 423/852] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:11.356914 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcd3321-4a6c-41f8-a5fe-03c2bb6436c7 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:11.357464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcd3321-4a6c-41f8-a5fe-03c2bb6436c7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:11.358239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcd3321-4a6c-41f8-a5fe-03c2bb6436c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:11.369983 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef862d2-cc16-4d69-8dd8-6a288c1b1f97 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:11.383138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-fdcd3321-4a6c-41f8-a5fe-03c2bb6436c7 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:11.385349 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fdcd3321-4a6c-41f8-a5fe-03c2bb6436c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:11.408449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce1ec51-3114-4176-aff0-ef2ae8f82d9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:11.409084 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce1ec51-3114-4176-aff0-ef2ae8f82d9b None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:11.409366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce1ec51-3114-4176-aff0-ef2ae8f82d9b None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:11.409638 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce1ec51-3114-4176-aff0-ef2ae8f82d9b None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:11.410274 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce1ec51-3114-4176-aff0-ef2ae8f82d9b None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:11.411094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce1ec51-3114-4176-aff0-ef2ae8f82d9b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:11.412514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce1ec51-3114-4176-aff0-ef2ae8f82d9b None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:11.418264 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 430/853] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:28:11 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:11.418500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce1ec51-3114-4176-aff0-ef2ae8f82d9b None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:11.464092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 424/854] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:11 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:11.465251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcd3321-4a6c-41f8-a5fe-03c2bb6436c7 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:11.551675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:11.555079 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce1ec51-3114-4176-aff0-ef2ae8f82d9b None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:11.582165 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:11.582355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcd3321-4a6c-41f8-a5fe-03c2bb6436c7 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:11.594410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f19d43a-bb9d-4051-bdc0-6bc828ced4a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:11.595443 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f19d43a-bb9d-4051-bdc0-6bc828ced4a1 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:11.595931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f19d43a-bb9d-4051-bdc0-6bc828ced4a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:11.596398 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f19d43a-bb9d-4051-bdc0-6bc828ced4a1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:11.603695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f19d43a-bb9d-4051-bdc0-6bc828ced4a1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:11.605142 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f19d43a-bb9d-4051-bdc0-6bc828ced4a1 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:11.617342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-5f19d43a-bb9d-4051-bdc0-6bc828ced4a1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:11.617342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:11.617342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:11.617342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:11.617342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:11.617342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:11.617342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:11.617342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:11.617342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:11.617342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:11.617342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:11.617342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:11.617342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:28:11.617342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:28:11.617342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:28:11.618324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:28:11.618324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:11.618324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:11.618324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:11.618324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:11.618324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:11.618324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:11.618324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:11.618324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:11.618324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:11.618324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:11.618324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:11.618324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:11.618324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:11.618324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:11.618324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:11.618324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:11.619297 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:28:11.619297 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:28:11.619297 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:28:11.619297 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:28:11.619297 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:28:11.619297 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:28:11.619297 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:11.619297 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:11.623460 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 431/855] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:28:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:11.624445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f19d43a-bb9d-4051-bdc0-6bc828ced4a1 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:11.633127 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb74b05-22c5-422f-be6d-372ccd7f98d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:11.633994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb74b05-22c5-422f-be6d-372ccd7f98d8 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:11.634473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb74b05-22c5-422f-be6d-372ccd7f98d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:11.634943 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb74b05-22c5-422f-be6d-372ccd7f98d8 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:11.636294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb74b05-22c5-422f-be6d-372ccd7f98d8 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:11.636782 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb74b05-22c5-422f-be6d-372ccd7f98d8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:11.638805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb74b05-22c5-422f-be6d-372ccd7f98d8 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:11.646558 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 425/856] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:28:11 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:11.749110 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f19d43a-bb9d-4051-bdc0-6bc828ced4a1 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:11.755586 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb74b05-22c5-422f-be6d-372ccd7f98d8 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:11.799246 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4db59aa-9fbd-443f-8bbb-eec093b35ff0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:11.800419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4db59aa-9fbd-443f-8bbb-eec093b35ff0 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:11.801784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4db59aa-9fbd-443f-8bbb-eec093b35ff0 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:11.802255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4db59aa-9fbd-443f-8bbb-eec093b35ff0 None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:11.803205 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4db59aa-9fbd-443f-8bbb-eec093b35ff0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:11.803719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4db59aa-9fbd-443f-8bbb-eec093b35ff0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:11.805376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4db59aa-9fbd-443f-8bbb-eec093b35ff0 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:11.823287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 432/857] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:28:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:11.830411 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4db59aa-9fbd-443f-8bbb-eec093b35ff0 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:11.879859 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:11.880587 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb74b05-22c5-422f-be6d-372ccd7f98d8 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:11.917374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422f4a40-121a-4604-8925-cdbef9f8b125 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:11.918013 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422f4a40-121a-4604-8925-cdbef9f8b125 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:11.918291 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422f4a40-121a-4604-8925-cdbef9f8b125 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:11.918674 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422f4a40-121a-4604-8925-cdbef9f8b125 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:11.930242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422f4a40-121a-4604-8925-cdbef9f8b125 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:28:11.935311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-422f4a40-121a-4604-8925-cdbef9f8b125 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:28:11.935311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:11.935311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:11.935311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:11.935311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:11.935311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:11.935311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:11.935311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:11.935311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:11.935311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:11.935311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:11.935311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:11.935311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:28:11.935311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:28:11.935311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:28:11.935311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:28:11.936252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:28:11.936252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:28:11.936252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:28:11.936252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:28:11.936252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:11.936252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:11.936252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:11.936252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:11.936252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:11.936252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:11.936252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:11.936252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:11.936252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:11.936252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:11.936252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:11.936252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:11.937122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:11.937122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:11.937122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:11.937122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:11.937122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:28:11.937122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:28:11.937122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:28:11.937122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:28:11.937122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:28:11.937122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:28:11.937122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:28:11.937122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:11.937122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 426/858] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:28:11 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:11.943957 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4db59aa-9fbd-443f-8bbb-eec093b35ff0 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:11.945008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422f4a40-121a-4604-8925-cdbef9f8b125 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:11.950415 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:11.984790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66686c9e-ee49-455b-a7aa-ccf96c18ad08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:11.985504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66686c9e-ee49-455b-a7aa-ccf96c18ad08 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:11.985794 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66686c9e-ee49-455b-a7aa-ccf96c18ad08 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:11.986066 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66686c9e-ee49-455b-a7aa-ccf96c18ad08 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:11.986902 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66686c9e-ee49-455b-a7aa-ccf96c18ad08 None admin] RBAC: Adding query filter params (role.id=89c4b3a971224d468d33d2dc411230a2, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=aa30da9972e140d7905c522576681429) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:11.987252 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66686c9e-ee49-455b-a7aa-ccf96c18ad08 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:11.988685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66686c9e-ee49-455b-a7aa-ccf96c18ad08 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:11.995566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 433/859] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:28:11 2022] GET /identity/v3/role_assignments?user.id=aa30da9972e140d7905c522576681429&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=89c4b3a971224d468d33d2dc411230a2 => generated 608 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:12.076173 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422f4a40-121a-4604-8925-cdbef9f8b125 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:12.121744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b7e830-d0d5-4cbd-b19e-1b479c618696 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:12.122742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b7e830-d0d5-4cbd-b19e-1b479c618696 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:12.123257 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b7e830-d0d5-4cbd-b19e-1b479c618696 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:12.124276 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b7e830-d0d5-4cbd-b19e-1b479c618696 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:12.125387 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b7e830-d0d5-4cbd-b19e-1b479c618696 None admin] RBAC: Adding query filter params (name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:12.126225 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b7e830-d0d5-4cbd-b19e-1b479c618696 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:12.127896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b7e830-d0d5-4cbd-b19e-1b479c618696 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:12.138280 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 427/860] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:28:11 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:12.145760 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66686c9e-ee49-455b-a7aa-ccf96c18ad08 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:12.260730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66686c9e-ee49-455b-a7aa-ccf96c18ad08 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:12.271273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:12.298237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19785dd6-e892-48b1-a3e0-b9fcb8a1eb51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:12.299279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19785dd6-e892-48b1-a3e0-b9fcb8a1eb51 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:12.299769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19785dd6-e892-48b1-a3e0-b9fcb8a1eb51 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:12.300265 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19785dd6-e892-48b1-a3e0-b9fcb8a1eb51 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:12.312818 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19785dd6-e892-48b1-a3e0-b9fcb8a1eb51 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:12.315926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19785dd6-e892-48b1-a3e0-b9fcb8a1eb51 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:12.343626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-19785dd6-e892-48b1-a3e0-b9fcb8a1eb51 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:12.343626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:12.343626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:12.343626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:12.343626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:12.343626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:12.343626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:12.343626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:12.343626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:12.343626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:12.343626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:12.343626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:12.343626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:12.343626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:12.343626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:12.343626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:12.345921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:12.345921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:12.345921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:12.345921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:12.345921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:12.345921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:12.345921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:12.345921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:12.345921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:12.345921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:12.345921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:12.345921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:12.345921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:12.345921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:12.345921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:12.345921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:12.345921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:12.348864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:12.348864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:12.348864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:12.351773 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 434/861] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:12.361073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b7e830-d0d5-4cbd-b19e-1b479c618696 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:12.472582 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:12.539649 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b7e830-d0d5-4cbd-b19e-1b479c618696 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:12.586788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3171d5-a9d5-4d28-9276-47933b5c8c0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:12.587432 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3171d5-a9d5-4d28-9276-47933b5c8c0e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:12.587710 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3171d5-a9d5-4d28-9276-47933b5c8c0e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:12.587978 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3171d5-a9d5-4d28-9276-47933b5c8c0e None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:12.588618 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3171d5-a9d5-4d28-9276-47933b5c8c0e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:12.588919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3171d5-a9d5-4d28-9276-47933b5c8c0e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:12.590273 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3171d5-a9d5-4d28-9276-47933b5c8c0e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:12.603263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 428/862] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:12.611960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19785dd6-e892-48b1-a3e0-b9fcb8a1eb51 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:12.750883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19785dd6-e892-48b1-a3e0-b9fcb8a1eb51 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:12.789024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573dffba-77de-4e22-8717-7575db26f5a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:12.791104 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573dffba-77de-4e22-8717-7575db26f5a2 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:12.791626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573dffba-77de-4e22-8717-7575db26f5a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:12.792102 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573dffba-77de-4e22-8717-7575db26f5a2 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:12.802723 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573dffba-77de-4e22-8717-7575db26f5a2 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:12.804521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573dffba-77de-4e22-8717-7575db26f5a2 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:12.822327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-573dffba-77de-4e22-8717-7575db26f5a2 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 03:28:12.822327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:12.822327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:12.822327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:12.822327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:12.822327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:12.822327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:12.822327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:12.822327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:12.822327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:12.822327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:12.822327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:12.822327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:28:12.822327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:28:12.822327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:28:12.822327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:28:12.823427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:12.823427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:12.823427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:28:12.823427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:12.823427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:28:12.823427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:12.823427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:12.823427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:12.823427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:12.823427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:12.823427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:12.823427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:12.823427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:12.823427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:12.823427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:12.823427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:12.823427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:12.823427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:12.824382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:12.824382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:12.824382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:28:12.824382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:28:12.824382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:28:12.824382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:28:12.824382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:28:12.824382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:28:12.824382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 03:28:12.824382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:12.831197 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 435/863] 10.222.0.49 () {56 vars in 1117 bytes} [Fri Sep 16 03:28:12 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:12.840704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3171d5-a9d5-4d28-9276-47933b5c8c0e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:12.951614 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3171d5-a9d5-4d28-9276-47933b5c8c0e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:12.981067 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:12.998577 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241b118b-12ac-4bde-ad77-d367e0545589 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:12.998577 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241b118b-12ac-4bde-ad77-d367e0545589 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:12.998577 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241b118b-12ac-4bde-ad77-d367e0545589 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:12.998577 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241b118b-12ac-4bde-ad77-d367e0545589 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:12.999043 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241b118b-12ac-4bde-ad77-d367e0545589 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:12.999043 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241b118b-12ac-4bde-ad77-d367e0545589 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:13.001024 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241b118b-12ac-4bde-ad77-d367e0545589 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:13.026813 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 429/864] 10.222.0.49 () {56 vars in 1149 bytes} [Fri Sep 16 03:28:12 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:13.033432 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573dffba-77de-4e22-8717-7575db26f5a2 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:13.060702 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:13.159913 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573dffba-77de-4e22-8717-7575db26f5a2 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:13.203129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab49e0d4-3aa7-4b6a-b073-defa6859f549 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:13.203796 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab49e0d4-3aa7-4b6a-b073-defa6859f549 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:13.204064 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab49e0d4-3aa7-4b6a-b073-defa6859f549 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:13.204340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab49e0d4-3aa7-4b6a-b073-defa6859f549 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:13.290052 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab49e0d4-3aa7-4b6a-b073-defa6859f549 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:13.292092 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab49e0d4-3aa7-4b6a-b073-defa6859f549 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:13.318525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:13.424183 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 436/865] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:28:13 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 03:28:13.429827 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241b118b-12ac-4bde-ad77-d367e0545589 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:13.547468 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241b118b-12ac-4bde-ad77-d367e0545589 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:13.584153 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abe91f7-0142-46e7-8b7f-c079a3086b5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:13.585212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abe91f7-0142-46e7-8b7f-c079a3086b5f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:13.585748 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abe91f7-0142-46e7-8b7f-c079a3086b5f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:13.586217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abe91f7-0142-46e7-8b7f-c079a3086b5f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:13.598186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abe91f7-0142-46e7-8b7f-c079a3086b5f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:13.600235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abe91f7-0142-46e7-8b7f-c079a3086b5f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:13.616186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-0abe91f7-0142-46e7-8b7f-c079a3086b5f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:13.616186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:13.616186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:13.616186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:13.616186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:13.616186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:13.616186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:13.616186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:13.616186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:13.616186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:13.616186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:13.616186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:13.616186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:13.616186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:13.616186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:13.616186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:13.617213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:13.617213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:13.617213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:13.617213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:13.617213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:13.617213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:13.617213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:13.617213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:13.617213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:13.617213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:13.617213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:13.617213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:13.617213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:13.617213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:13.617213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:13.617213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:13.617213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:13.618136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:13.618136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:13.618136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:13.620275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 430/866] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:13.626321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab49e0d4-3aa7-4b6a-b073-defa6859f549 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:13.635230 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8134c4cc-6a34-4c86-a610-d9d4fd07c811 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:13.635838 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8134c4cc-6a34-4c86-a610-d9d4fd07c811 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:13.636319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8134c4cc-6a34-4c86-a610-d9d4fd07c811 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:13.638116 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 431/867] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:28:13.644612 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5843a53-ab4f-4fe2-b989-10c3870977ed None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:13.645172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5843a53-ab4f-4fe2-b989-10c3870977ed None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:13.645668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5843a53-ab4f-4fe2-b989-10c3870977ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:13.665192 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a5843a53-ab4f-4fe2-b989-10c3870977ed None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:13.666887 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a5843a53-ab4f-4fe2-b989-10c3870977ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:13.697357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:13.751278 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 432/868] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:13 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 03:28:13.758752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab49e0d4-3aa7-4b6a-b073-defa6859f549 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:13.788654 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:13.809201 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b30955c-6857-465e-a166-a159a5f358a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:13.809201 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b30955c-6857-465e-a166-a159a5f358a6 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:13.809201 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b30955c-6857-465e-a166-a159a5f358a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:13.809201 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b30955c-6857-465e-a166-a159a5f358a6 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:13.809589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b30955c-6857-465e-a166-a159a5f358a6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:13.809589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b30955c-6857-465e-a166-a159a5f358a6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:13.809589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b30955c-6857-465e-a166-a159a5f358a6 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:13.821427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 437/869] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:13.826966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5843a53-ab4f-4fe2-b989-10c3870977ed None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:13.932581 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5843a53-ab4f-4fe2-b989-10c3870977ed None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:13.969230 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1be116-27e1-4f37-bd53-df269a07f992 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:13.970453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1be116-27e1-4f37-bd53-df269a07f992 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:13.970977 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1be116-27e1-4f37-bd53-df269a07f992 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:13.971466 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1be116-27e1-4f37-bd53-df269a07f992 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:13.979029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1be116-27e1-4f37-bd53-df269a07f992 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:13.980889 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1be116-27e1-4f37-bd53-df269a07f992 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:13.994305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-fa1be116-27e1-4f37-bd53-df269a07f992 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:13.994305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:13.994305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:13.994305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:13.994305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:13.994305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:13.994305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:13.994305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:13.994305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:13.994305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:13.994305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:13.994305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:13.994305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:28:13.994305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:28:13.994305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:28:13.995292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:28:13.995292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:13.995292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:13.995292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:13.995292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:13.995292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:13.995292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:13.995292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:13.995292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:13.995292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:13.995292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:13.995292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:13.995292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:13.995292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:13.995292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:13.995292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:13.995292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:13.996181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:28:13.996181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:28:13.996181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:28:13.996181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:28:13.996181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:28:13.996181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:28:13.996181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:13.996181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:13.998280 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 433/870] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:28:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:14.004344 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b30955c-6857-465e-a166-a159a5f358a6 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:14.023612 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1be116-27e1-4f37-bd53-df269a07f992 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:14.054589 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:14.120892 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b30955c-6857-465e-a166-a159a5f358a6 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:14.134605 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1be116-27e1-4f37-bd53-df269a07f992 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:14.159980 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27eb51e8-2fe0-4c26-9fed-a926b766cc54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:14.160854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eb51e8-2fe0-4c26-9fed-a926b766cc54 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:14.161133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eb51e8-2fe0-4c26-9fed-a926b766cc54 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:14.161408 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eb51e8-2fe0-4c26-9fed-a926b766cc54 None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:14.162104 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27eb51e8-2fe0-4c26-9fed-a926b766cc54 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:14.162411 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27eb51e8-2fe0-4c26-9fed-a926b766cc54 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:14.163949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27eb51e8-2fe0-4c26-9fed-a926b766cc54 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:14.171571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc77a8d-8cea-4d83-98c8-d5c4d21ac565 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:14.172797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc77a8d-8cea-4d83-98c8-d5c4d21ac565 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:14.173076 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc77a8d-8cea-4d83-98c8-d5c4d21ac565 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:14.173377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc77a8d-8cea-4d83-98c8-d5c4d21ac565 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:14.174101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc77a8d-8cea-4d83-98c8-d5c4d21ac565 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:14.175487 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc77a8d-8cea-4d83-98c8-d5c4d21ac565 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:14.177117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 438/871] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:28:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:14.178347 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7744518-21a8-4fd7-8ae5-5212a45462ee None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:14.178631 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7744518-21a8-4fd7-8ae5-5212a45462ee None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:14.178934 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7744518-21a8-4fd7-8ae5-5212a45462ee None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:14.180262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 439/872] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:14.184053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7744518-21a8-4fd7-8ae5-5212a45462ee None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:14.228735 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:14.251830 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 434/873] 10.222.0.49 () {60 vars in 1141 bytes} [Fri Sep 16 03:28:14 2022] POST /identity/v3/services => generated 269 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:28:14.253185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68892cf7-ae0b-4174-b37e-d73d013509de None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:14.253570 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68892cf7-ae0b-4174-b37e-d73d013509de None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:14.253995 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68892cf7-ae0b-4174-b37e-d73d013509de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:14.276694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-68892cf7-ae0b-4174-b37e-d73d013509de None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:14.278256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-68892cf7-ae0b-4174-b37e-d73d013509de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:14.306764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7744518-21a8-4fd7-8ae5-5212a45462ee None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:14.356908 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299da00c-8905-4b87-aeff-44ffc3b0e252 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:14.357570 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299da00c-8905-4b87-aeff-44ffc3b0e252 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:14.357852 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299da00c-8905-4b87-aeff-44ffc3b0e252 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:14.358124 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299da00c-8905-4b87-aeff-44ffc3b0e252 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:14.360348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299da00c-8905-4b87-aeff-44ffc3b0e252 None admin] RBAC: Adding query filter params (role.id=974d839067e9411aa4cb850cee33f6ae, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=d40810f61ead4c3da414194b325da80a) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:14.360765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299da00c-8905-4b87-aeff-44ffc3b0e252 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:14.362511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299da00c-8905-4b87-aeff-44ffc3b0e252 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:14.369884 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 440/874] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:28:14 2022] GET /identity/v3/role_assignments?user.id=d40810f61ead4c3da414194b325da80a&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=974d839067e9411aa4cb850cee33f6ae => generated 608 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:14.370232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 435/875] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:14 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:14.476850 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75554dc-a4fd-42a3-9eea-41ff52ced7bf None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:14.477161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75554dc-a4fd-42a3-9eea-41ff52ced7bf None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:14.477432 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75554dc-a4fd-42a3-9eea-41ff52ced7bf None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:14.478992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 441/876] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:14.486861 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebe09fe-afb7-402b-9f8c-ab9aba41f42c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:14.488398 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebe09fe-afb7-402b-9f8c-ab9aba41f42c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:14.489165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebe09fe-afb7-402b-9f8c-ab9aba41f42c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:14.509313 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-4ebe09fe-afb7-402b-9f8c-ab9aba41f42c None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:14.510880 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4ebe09fe-afb7-402b-9f8c-ab9aba41f42c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:14.524291 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:14.589467 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 436/877] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:14 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:14.657153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75554dc-a4fd-42a3-9eea-41ff52ced7bf None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:14.771702 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:14.775606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75554dc-a4fd-42a3-9eea-41ff52ced7bf None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:14.809660 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a29aa92-9f04-409c-a3e9-ad4e8c4a51fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:14.810269 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a29aa92-9f04-409c-a3e9-ad4e8c4a51fb None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:14.810545 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a29aa92-9f04-409c-a3e9-ad4e8c4a51fb None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:14.810812 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a29aa92-9f04-409c-a3e9-ad4e8c4a51fb None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:14.811537 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a29aa92-9f04-409c-a3e9-ad4e8c4a51fb None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:14.813430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a29aa92-9f04-409c-a3e9-ad4e8c4a51fb None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:14.878799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebe09fe-afb7-402b-9f8c-ab9aba41f42c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:14.915762 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 442/878] 10.222.0.49 () {60 vars in 1141 bytes} [Fri Sep 16 03:28:14 2022] POST /identity/v3/services => generated 251 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:28:14.999092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebe09fe-afb7-402b-9f8c-ab9aba41f42c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:15.054201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b90ba5a-02c4-46ee-9b8d-6f16ae061357 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:15.063782 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b90ba5a-02c4-46ee-9b8d-6f16ae061357 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:15.064114 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b90ba5a-02c4-46ee-9b8d-6f16ae061357 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:15.064340 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b90ba5a-02c4-46ee-9b8d-6f16ae061357 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:15.064962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b90ba5a-02c4-46ee-9b8d-6f16ae061357 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:15.066389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b90ba5a-02c4-46ee-9b8d-6f16ae061357 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:15.083158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-2b90ba5a-02c4-46ee-9b8d-6f16ae061357 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 03:28:15.083158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:15.083158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:15.083158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:15.083158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:15.083158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:15.083158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:15.083158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:15.083158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 03:28:15.083158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:15.083158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:15.083158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:15.083158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:15.083158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:15.083158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:15.083158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:15.083158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:15.084275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:15.084275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:15.084275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:15.084275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:15.084275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:15.084275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:15.084275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:15.084275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:15.084275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:15.084275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:15.084275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:15.084275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:15.084275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:15.084275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:15.084275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:15.084275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:15.085125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:15.085125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:15.085125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:15.085125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:15.085125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:15.085125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:15.085125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:15.085125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:15.085125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:15.085125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:15.085125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:15.085125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:15.085125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 03:28:15.085125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:15.085125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 437/879] 10.222.0.49 () {56 vars in 1126 bytes} [Fri Sep 16 03:28:14 2022] GET /identity/v3/services/object-store => generated 93 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:15.092080 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a29aa92-9f04-409c-a3e9-ad4e8c4a51fb None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:15.198671 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a29aa92-9f04-409c-a3e9-ad4e8c4a51fb None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:15.233956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153e3974-0b68-4aa4-b48d-4a99c1b3061f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:15.235324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153e3974-0b68-4aa4-b48d-4a99c1b3061f None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:15.235612 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153e3974-0b68-4aa4-b48d-4a99c1b3061f None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:15.235883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153e3974-0b68-4aa4-b48d-4a99c1b3061f None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:15.236517 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153e3974-0b68-4aa4-b48d-4a99c1b3061f None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:15.236864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153e3974-0b68-4aa4-b48d-4a99c1b3061f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:15.238238 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153e3974-0b68-4aa4-b48d-4a99c1b3061f None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:15.243787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 443/880] 10.222.0.49 () {56 vars in 1122 bytes} [Fri Sep 16 03:28:15 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:15.249412 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b90ba5a-02c4-46ee-9b8d-6f16ae061357 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:15.253446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:15.357598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b90ba5a-02c4-46ee-9b8d-6f16ae061357 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:15.396954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb4b129-450a-404f-bc91-0d4072147d1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:15.397561 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb4b129-450a-404f-bc91-0d4072147d1b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:15.397840 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb4b129-450a-404f-bc91-0d4072147d1b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:15.398120 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb4b129-450a-404f-bc91-0d4072147d1b None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:15.398748 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb4b129-450a-404f-bc91-0d4072147d1b None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:15.399090 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb4b129-450a-404f-bc91-0d4072147d1b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:15.400505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb4b129-450a-404f-bc91-0d4072147d1b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:15.405638 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 438/881] 10.222.0.49 () {56 vars in 1122 bytes} [Fri Sep 16 03:28:15 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:16.582311 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977265df-74f9-4e53-94cc-221e8e7e0fb4 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:16.583037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977265df-74f9-4e53-94cc-221e8e7e0fb4 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:16.583037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977265df-74f9-4e53-94cc-221e8e7e0fb4 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:16.584352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 444/882] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:16.589703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf668228-23b8-4131-b710-0f460f42cf38 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:16.590194 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf668228-23b8-4131-b710-0f460f42cf38 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:16.590363 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf668228-23b8-4131-b710-0f460f42cf38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:16.610110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-bf668228-23b8-4131-b710-0f460f42cf38 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:16.611503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-bf668228-23b8-4131-b710-0f460f42cf38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:16.680722 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:16.686299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 439/883] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:16 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:16.745353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2ddc65-8874-41af-9a85-103260482ccb None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:16.745734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2ddc65-8874-41af-9a85-103260482ccb None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:16.746083 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2ddc65-8874-41af-9a85-103260482ccb None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:16.747779 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 445/884] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:16.753145 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a9de31-20fd-4d05-a495-ea1031ee260d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:16.753503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a9de31-20fd-4d05-a495-ea1031ee260d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:16.753857 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a9de31-20fd-4d05-a495-ea1031ee260d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:16.772623 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-88a9de31-20fd-4d05-a495-ea1031ee260d None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:16.773818 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-88a9de31-20fd-4d05-a495-ea1031ee260d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:16.842198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 440/885] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:16 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:16.954292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2ddc65-8874-41af-9a85-103260482ccb None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:16.984074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:17.067410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2ddc65-8874-41af-9a85-103260482ccb None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:17.103734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1368d55d-f627-4430-8efc-11efa8b87131 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:17.104311 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1368d55d-f627-4430-8efc-11efa8b87131 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:17.104595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1368d55d-f627-4430-8efc-11efa8b87131 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:17.104869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1368d55d-f627-4430-8efc-11efa8b87131 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:17.105472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1368d55d-f627-4430-8efc-11efa8b87131 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:17.106929 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1368d55d-f627-4430-8efc-11efa8b87131 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:17.117590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-1368d55d-f627-4430-8efc-11efa8b87131 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 16 03:28:17.117590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:17.117590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:17.117590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:17.117590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:17.117590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:17.117590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:17.117590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:17.117590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 16 03:28:17.117590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:17.117590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:17.117590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:17.117590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:17.117590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:17.117590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:17.117590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:17.118632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:17.118632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:17.118632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:17.118632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:17.118632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:17.118632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:17.118632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:17.118632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:17.118632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:17.118632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:17.118632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:17.118632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:17.118632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:17.118632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:17.118632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:17.118632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:17.118632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:17.119576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:17.119576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:17.119576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:17.119576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:17.119576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:17.119576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:17.119576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:17.119576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:17.119576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:17.119576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:17.119576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:17.119576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:17.119576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 16 03:28:17.119576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:17.119576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 446/886] 10.222.0.49 () {56 vars in 1132 bytes} [Fri Sep 16 03:28:16 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:17.127693 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a9de31-20fd-4d05-a495-ea1031ee260d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:17.144273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1368d55d-f627-4430-8efc-11efa8b87131 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:17.244766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:17.253029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a9de31-20fd-4d05-a495-ea1031ee260d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:17.259966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1368d55d-f627-4430-8efc-11efa8b87131 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:17.285019 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:17.293483 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51437a55-a96e-42a3-9d40-025a9d5da27a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:17.296733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51437a55-a96e-42a3-9d40-025a9d5da27a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:17.296733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51437a55-a96e-42a3-9d40-025a9d5da27a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:17.296733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51437a55-a96e-42a3-9d40-025a9d5da27a None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:17.296733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51437a55-a96e-42a3-9d40-025a9d5da27a None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:17.296733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51437a55-a96e-42a3-9d40-025a9d5da27a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:17.297700 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15d6ee1-32d8-45e9-ae94-0d51366ae5b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:17.297700 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15d6ee1-32d8-45e9-ae94-0d51366ae5b9 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:17.297700 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15d6ee1-32d8-45e9-ae94-0d51366ae5b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:17.297918 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51437a55-a96e-42a3-9d40-025a9d5da27a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:17.297988 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15d6ee1-32d8-45e9-ae94-0d51366ae5b9 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:17.310033 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 441/887] 10.222.0.49 () {56 vars in 1126 bytes} [Fri Sep 16 03:28:17 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:17.310033 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5db201-5eef-49bc-8546-5eb0a9cb7f31 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:17.310033 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5db201-5eef-49bc-8546-5eb0a9cb7f31 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:17.310033 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5db201-5eef-49bc-8546-5eb0a9cb7f31 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:17.310033 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 442/888] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:17.310033 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5db201-5eef-49bc-8546-5eb0a9cb7f31 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:17.317183 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e15d6ee1-32d8-45e9-ae94-0d51366ae5b9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:28:17.323128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-e15d6ee1-32d8-45e9-ae94-0d51366ae5b9 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 03:28:17.323128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:17.323128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:17.323128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:17.323128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:17.323128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:17.323128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:17.323128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:17.323128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:17.323128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:17.323128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:17.323128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:17.323128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:28:17.323128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:28:17.323128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:28:17.323128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:28:17.324551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:28:17.324551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:28:17.324551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:28:17.324551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:28:17.324551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:17.324551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:17.324551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:17.324551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:17.324551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:17.324551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:17.324551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:17.324551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:17.324551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:17.324551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:17.324551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:17.324551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:17.327752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:17.327752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:17.327752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:17.327752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:17.327752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:28:17.327752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:28:17.327752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:28:17.327752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:28:17.327752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:28:17.327752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:28:17.327752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 03:28:17.327752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:17.327752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 447/889] 10.222.0.49 () {56 vars in 1120 bytes} [Fri Sep 16 03:28:17 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:17.327752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f3397f-b182-4735-aa74-fbafa4a7be72 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:17.329627 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f3397f-b182-4735-aa74-fbafa4a7be72 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:17.329627 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f3397f-b182-4735-aa74-fbafa4a7be72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:17.346333 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-11f3397f-b182-4735-aa74-fbafa4a7be72 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:17.347779 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-11f3397f-b182-4735-aa74-fbafa4a7be72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:17.417939 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 448/890] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:17 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:17.418723 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f3397f-b182-4735-aa74-fbafa4a7be72 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:17.437548 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5db201-5eef-49bc-8546-5eb0a9cb7f31 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:17.474572 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8d22dd-3bbf-48b5-b958-625f1a330681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:17.475235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8d22dd-3bbf-48b5-b958-625f1a330681 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:17.475514 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8d22dd-3bbf-48b5-b958-625f1a330681 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:17.475786 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8d22dd-3bbf-48b5-b958-625f1a330681 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:17.476444 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8d22dd-3bbf-48b5-b958-625f1a330681 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:17.477060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8d22dd-3bbf-48b5-b958-625f1a330681 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:17.478290 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8d22dd-3bbf-48b5-b958-625f1a330681 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:17.489130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 443/891] 10.222.0.49 () {56 vars in 1126 bytes} [Fri Sep 16 03:28:17 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:17.490818 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8d22dd-3bbf-48b5-b958-625f1a330681 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:17.513264 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:17.543455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f3397f-b182-4735-aa74-fbafa4a7be72 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:17.568206 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:17.594615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbb170f-24ae-4a32-bf28-1d966e2d5f07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:17.596105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbb170f-24ae-4a32-bf28-1d966e2d5f07 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:17.596560 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbb170f-24ae-4a32-bf28-1d966e2d5f07 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:17.597127 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbb170f-24ae-4a32-bf28-1d966e2d5f07 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:17.598375 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbb170f-24ae-4a32-bf28-1d966e2d5f07 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:17.598830 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbb170f-24ae-4a32-bf28-1d966e2d5f07 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:17.600752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbb170f-24ae-4a32-bf28-1d966e2d5f07 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:17.612459 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 449/892] 10.222.0.49 () {56 vars in 1115 bytes} [Fri Sep 16 03:28:17 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:17.618205 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbb170f-24ae-4a32-bf28-1d966e2d5f07 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:17.625950 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8d22dd-3bbf-48b5-b958-625f1a330681 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:17.665587 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32af86b9-8af0-4f68-b31d-2aacce8af59f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:17.666530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32af86b9-8af0-4f68-b31d-2aacce8af59f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:17.666811 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32af86b9-8af0-4f68-b31d-2aacce8af59f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:17.667111 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32af86b9-8af0-4f68-b31d-2aacce8af59f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:17.667914 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32af86b9-8af0-4f68-b31d-2aacce8af59f None admin] RBAC: Adding query filter params (interface=public, service_id=d2a8c57b203046a9bd056d4522b08c8e, region_id=RegionOne) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:17.668390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32af86b9-8af0-4f68-b31d-2aacce8af59f None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:17.669683 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32af86b9-8af0-4f68-b31d-2aacce8af59f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:17.677609 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 444/893] 10.222.0.49 () {56 vars in 1251 bytes} [Fri Sep 16 03:28:17 2022] GET /identity/v3/endpoints?service_id=d2a8c57b203046a9bd056d4522b08c8e&interface=public®ion_id=RegionOne => generated 194 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:17.679117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbd3f52-32ba-44e9-98ec-20acd4b5a6ec None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:17.679628 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbd3f52-32ba-44e9-98ec-20acd4b5a6ec None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:17.680132 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbd3f52-32ba-44e9-98ec-20acd4b5a6ec None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:17.681614 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 445/894] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:17.682500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbd3f52-32ba-44e9-98ec-20acd4b5a6ec None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:17.725353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbb170f-24ae-4a32-bf28-1d966e2d5f07 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:17.757684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd4888d-80eb-40d8-b69a-4ee36a991b0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:17.758307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd4888d-80eb-40d8-b69a-4ee36a991b0d None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:17.758583 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd4888d-80eb-40d8-b69a-4ee36a991b0d None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:17.758854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd4888d-80eb-40d8-b69a-4ee36a991b0d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:17.767341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:17.769439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd4888d-80eb-40d8-b69a-4ee36a991b0d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:17.770858 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd4888d-80eb-40d8-b69a-4ee36a991b0d None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:17.782328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbd3f52-32ba-44e9-98ec-20acd4b5a6ec None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:17.785093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-9dd4888d-80eb-40d8-b69a-4ee36a991b0d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:17.785093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:17.785093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:17.785093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:17.785093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:17.785093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:17.785093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:17.785093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:17.785093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:17.785093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:17.785093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:17.785093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:17.785093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:17.785093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:17.785093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:17.785093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:17.786118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:17.786118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:17.786118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:17.786118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:17.786118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:17.786118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:17.786118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:17.786118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:17.786118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:17.786118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:17.786118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:17.786118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:17.786118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:17.786118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:17.786118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:17.786118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:17.786118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:17.787769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:17.787769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:17.787769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:17.787769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 450/895] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:17.787769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c535a60e-cf39-493a-aecd-4c29879c1b28 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:17.787769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c535a60e-cf39-493a-aecd-4c29879c1b28 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:17.787769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c535a60e-cf39-493a-aecd-4c29879c1b28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:17.804723 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:17.806054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-c535a60e-cf39-493a-aecd-4c29879c1b28 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:17.807278 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c535a60e-cf39-493a-aecd-4c29879c1b28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:17.818044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2881b22-f5cb-4b48-972b-820edf757a42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:17.818649 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2881b22-f5cb-4b48-972b-820edf757a42 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:17.818949 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2881b22-f5cb-4b48-972b-820edf757a42 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:17.819226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2881b22-f5cb-4b48-972b-820edf757a42 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:17.819818 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2881b22-f5cb-4b48-972b-820edf757a42 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:17.821191 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2881b22-f5cb-4b48-972b-820edf757a42 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:17.826667 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 446/896] 10.222.0.49 () {56 vars in 1087 bytes} [Fri Sep 16 03:28:17 2022] GET /identity/v3/services => generated 1268 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:28:17.827462 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2881b22-f5cb-4b48-972b-820edf757a42 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:17.882991 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 451/897] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:17 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:17.883715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c535a60e-cf39-493a-aecd-4c29879c1b28 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:17.965192 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2881b22-f5cb-4b48-972b-820edf757a42 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:18.008724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73527f4a-b5c2-4cb6-aba5-d2ae09c67bb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:18.009520 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73527f4a-b5c2-4cb6-aba5-d2ae09c67bb6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:18.009892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73527f4a-b5c2-4cb6-aba5-d2ae09c67bb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:18.010264 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73527f4a-b5c2-4cb6-aba5-d2ae09c67bb6 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:18.011697 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73527f4a-b5c2-4cb6-aba5-d2ae09c67bb6 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:18.012366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73527f4a-b5c2-4cb6-aba5-d2ae09c67bb6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:18.023913 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c535a60e-cf39-493a-aecd-4c29879c1b28 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:18.024703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-73527f4a-b5c2-4cb6-aba5-d2ae09c67bb6 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 16 03:28:18.024703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:18.024703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:18.024703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:18.024703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:18.024703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:18.024703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:18.024703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:18.024703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 16 03:28:18.024703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:18.024703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:18.024703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:18.024703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:18.024703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:18.024703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:18.024703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:18.024703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:18.025898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:18.025898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:18.025898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:18.025898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:18.025898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:18.025898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:18.025898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:18.025898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:18.025898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:18.025898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:18.025898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:18.025898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:18.025898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:18.025898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:18.025898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:18.025898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:18.026837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:18.026837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:18.026837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:18.026837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:18.026837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:18.026837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:18.026837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:18.026837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:18.026837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:18.026837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:18.026837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:18.026837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:18.026837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 16 03:28:18.026837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:18.027755 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 447/898] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:28:17 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:18.035480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73527f4a-b5c2-4cb6-aba5-d2ae09c67bb6 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:18.051852 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:18.073090 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d39863-58c0-466e-b1f5-a30ec3a0d0ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:18.073964 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d39863-58c0-466e-b1f5-a30ec3a0d0ee None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:18.074248 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d39863-58c0-466e-b1f5-a30ec3a0d0ee None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:18.074524 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d39863-58c0-466e-b1f5-a30ec3a0d0ee None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:18.075258 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d39863-58c0-466e-b1f5-a30ec3a0d0ee None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:18.075570 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d39863-58c0-466e-b1f5-a30ec3a0d0ee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:18.078711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d39863-58c0-466e-b1f5-a30ec3a0d0ee None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:18.094372 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:18.100029 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 452/899] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:18.101174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d39863-58c0-466e-b1f5-a30ec3a0d0ee None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:18.167528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73527f4a-b5c2-4cb6-aba5-d2ae09c67bb6 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:18.204735 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3728eee8-aeda-486f-80c5-f25b85ef1016 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:18.205351 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3728eee8-aeda-486f-80c5-f25b85ef1016 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:18.205621 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3728eee8-aeda-486f-80c5-f25b85ef1016 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:18.205904 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3728eee8-aeda-486f-80c5-f25b85ef1016 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:18.206526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3728eee8-aeda-486f-80c5-f25b85ef1016 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:18.206829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3728eee8-aeda-486f-80c5-f25b85ef1016 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:18.208215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3728eee8-aeda-486f-80c5-f25b85ef1016 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:18.213450 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 448/900] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:28:18 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:18.214101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3728eee8-aeda-486f-80c5-f25b85ef1016 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:18.221258 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d39863-58c0-466e-b1f5-a30ec3a0d0ee None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:18.257873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9514e829-f065-4153-8ee4-ae19482e8079 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:18.258747 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9514e829-f065-4153-8ee4-ae19482e8079 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:18.259218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9514e829-f065-4153-8ee4-ae19482e8079 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:18.259633 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9514e829-f065-4153-8ee4-ae19482e8079 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:18.267445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9514e829-f065-4153-8ee4-ae19482e8079 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:18.269038 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9514e829-f065-4153-8ee4-ae19482e8079 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:18.280200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-9514e829-f065-4153-8ee4-ae19482e8079 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 03:28:18.280200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:18.280200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:18.280200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:18.280200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:18.280200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:18.280200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:18.280200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:18.280200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:18.280200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:18.280200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:18.280200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:18.280200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:28:18.280200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:28:18.280200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:28:18.280200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:28:18.281765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:18.281765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:18.281765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:28:18.281765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:18.281765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:28:18.281765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:18.281765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:18.281765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:18.281765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:18.281765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:18.281765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:18.281765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:18.281765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:18.281765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:18.281765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:18.281765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:18.281765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:18.281765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:18.282942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:18.282942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:18.282942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:28:18.282942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:28:18.282942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:28:18.282942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:28:18.282942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:28:18.282942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:28:18.282942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 03:28:18.282942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:18.284991 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 453/901] 10.222.0.49 () {56 vars in 1117 bytes} [Fri Sep 16 03:28:18 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:18.285920 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9514e829-f065-4153-8ee4-ae19482e8079 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:18.336717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3728eee8-aeda-486f-80c5-f25b85ef1016 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:18.337933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:18.353638 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:18.375455 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cba087-9dda-43ab-879a-81be13a82293 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:18.376452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cba087-9dda-43ab-879a-81be13a82293 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:18.376735 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cba087-9dda-43ab-879a-81be13a82293 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:18.377005 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cba087-9dda-43ab-879a-81be13a82293 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:18.377652 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cba087-9dda-43ab-879a-81be13a82293 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:18.379110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cba087-9dda-43ab-879a-81be13a82293 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:18.399948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9514e829-f065-4153-8ee4-ae19482e8079 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:18.434243 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81838a18-53b7-418e-8b0f-f8135501a94b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:18.434853 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81838a18-53b7-418e-8b0f-f8135501a94b None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:18.435166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81838a18-53b7-418e-8b0f-f8135501a94b None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:18.435439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81838a18-53b7-418e-8b0f-f8135501a94b None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:18.436061 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81838a18-53b7-418e-8b0f-f8135501a94b None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:18.436363 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81838a18-53b7-418e-8b0f-f8135501a94b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:18.438550 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81838a18-53b7-418e-8b0f-f8135501a94b None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:18.443363 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 454/902] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:28:18 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:18.444137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81838a18-53b7-418e-8b0f-f8135501a94b None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:18.457924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 449/903] 10.222.0.49 () {60 vars in 1141 bytes} [Fri Sep 16 03:28:18 2022] POST /identity/v3/services => generated 241 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:28:18.458745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cba087-9dda-43ab-879a-81be13a82293 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:18.572208 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81838a18-53b7-418e-8b0f-f8135501a94b None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:18.581322 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:18.607685 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cba087-9dda-43ab-879a-81be13a82293 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:18.613191 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be962856-af66-4de9-9c9d-f53aa3a6d7c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:18.614539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be962856-af66-4de9-9c9d-f53aa3a6d7c9 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:18.615091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be962856-af66-4de9-9c9d-f53aa3a6d7c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:18.615700 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be962856-af66-4de9-9c9d-f53aa3a6d7c9 None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:18.616661 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be962856-af66-4de9-9c9d-f53aa3a6d7c9 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:18.617187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be962856-af66-4de9-9c9d-f53aa3a6d7c9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:18.618862 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be962856-af66-4de9-9c9d-f53aa3a6d7c9 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:18.640100 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 455/904] 10.222.0.49 () {56 vars in 1149 bytes} [Fri Sep 16 03:28:18 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:18.647753 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be962856-af66-4de9-9c9d-f53aa3a6d7c9 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:18.657000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8901a1-cdb2-46a3-965f-ec959dbb68e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:18.657747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8901a1-cdb2-46a3-965f-ec959dbb68e6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:18.658140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8901a1-cdb2-46a3-965f-ec959dbb68e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:18.658507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8901a1-cdb2-46a3-965f-ec959dbb68e6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:18.659765 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8901a1-cdb2-46a3-965f-ec959dbb68e6 None admin] RBAC: Adding query filter params (interface=public, service_id=4a6a5daf7d764f4ba423ec0e6c444b61, region_id=RegionOne) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:18.659765 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8901a1-cdb2-46a3-965f-ec959dbb68e6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:18.661457 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8901a1-cdb2-46a3-965f-ec959dbb68e6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:18.668644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 450/905] 10.222.0.49 () {56 vars in 1251 bytes} [Fri Sep 16 03:28:18 2022] GET /identity/v3/endpoints?service_id=4a6a5daf7d764f4ba423ec0e6c444b61&interface=public®ion_id=RegionOne => generated 194 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:18.674620 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8901a1-cdb2-46a3-965f-ec959dbb68e6 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:18.723382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:18.769903 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be962856-af66-4de9-9c9d-f53aa3a6d7c9 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:18.808262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ec494f-fa67-4cb9-87f8-1a874a0958a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:18.809414 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ec494f-fa67-4cb9-87f8-1a874a0958a3 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:18.809853 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ec494f-fa67-4cb9-87f8-1a874a0958a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:18.810305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ec494f-fa67-4cb9-87f8-1a874a0958a3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:18.811372 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8901a1-cdb2-46a3-965f-ec959dbb68e6 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:18.848172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4ec6d3-d7b3-42f4-81cd-5c98f992cded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:18.848892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4ec6d3-d7b3-42f4-81cd-5c98f992cded None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:18.849266 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4ec6d3-d7b3-42f4-81cd-5c98f992cded None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:18.849699 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4ec6d3-d7b3-42f4-81cd-5c98f992cded None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:18.850515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4ec6d3-d7b3-42f4-81cd-5c98f992cded None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:18.851998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4ec6d3-d7b3-42f4-81cd-5c98f992cded None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:18.854302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:18.866951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 451/906] 10.222.0.49 () {56 vars in 1087 bytes} [Fri Sep 16 03:28:18 2022] GET /identity/v3/services => generated 1497 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:28:18.905887 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ec494f-fa67-4cb9-87f8-1a874a0958a3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:18.907777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ec494f-fa67-4cb9-87f8-1a874a0958a3 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:19.031060 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 456/907] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:28:18 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 16 03:28:19.033486 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4ec6d3-d7b3-42f4-81cd-5c98f992cded None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:19.139370 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:19.145921 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4ec6d3-d7b3-42f4-81cd-5c98f992cded None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:19.180999 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dcaff9-5f16-4fc9-8cdd-e1b2a20fa5b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:19.181830 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dcaff9-5f16-4fc9-8cdd-e1b2a20fa5b5 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:19.182631 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dcaff9-5f16-4fc9-8cdd-e1b2a20fa5b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:19.183396 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dcaff9-5f16-4fc9-8cdd-e1b2a20fa5b5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:19.195108 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85dcaff9-5f16-4fc9-8cdd-e1b2a20fa5b5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:19.196629 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85dcaff9-5f16-4fc9-8cdd-e1b2a20fa5b5 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:19.212411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-85dcaff9-5f16-4fc9-8cdd-e1b2a20fa5b5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:19.212411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:19.212411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:19.212411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:19.212411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:19.212411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:19.212411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:19.212411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:19.212411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:19.212411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:19.212411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:19.212411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:19.212411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:19.212411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:19.212411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:19.212411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:19.213626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:19.213626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:19.213626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:19.213626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:19.213626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:19.213626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:19.213626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:19.213626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:19.213626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:19.213626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:19.213626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:19.213626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:19.213626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:19.213626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:19.213626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:19.213626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:19.213626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:19.214746 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:19.214746 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:19.214746 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:19.219092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 452/908] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:19.221219 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ec494f-fa67-4cb9-87f8-1a874a0958a3 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:19.288606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:19.330191 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ec494f-fa67-4cb9-87f8-1a874a0958a3 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:19.365839 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204a50f0-e93c-4979-8cd3-92c74aa123c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:19.366447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204a50f0-e93c-4979-8cd3-92c74aa123c5 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:19.366722 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204a50f0-e93c-4979-8cd3-92c74aa123c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:19.367024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204a50f0-e93c-4979-8cd3-92c74aa123c5 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:19.367788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204a50f0-e93c-4979-8cd3-92c74aa123c5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:19.368087 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204a50f0-e93c-4979-8cd3-92c74aa123c5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:19.369447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204a50f0-e93c-4979-8cd3-92c74aa123c5 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:19.385436 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 457/909] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:19.387697 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dcaff9-5f16-4fc9-8cdd-e1b2a20fa5b5 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:19.501351 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dcaff9-5f16-4fc9-8cdd-e1b2a20fa5b5 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:19.537223 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a479f5-0f1a-475f-80c4-bf72da6b0018 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:19.537855 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a479f5-0f1a-475f-80c4-bf72da6b0018 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:19.538131 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a479f5-0f1a-475f-80c4-bf72da6b0018 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:19.538403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a479f5-0f1a-475f-80c4-bf72da6b0018 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:19.545206 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a479f5-0f1a-475f-80c4-bf72da6b0018 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:19.547482 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a479f5-0f1a-475f-80c4-bf72da6b0018 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:19.559585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-e2a479f5-0f1a-475f-80c4-bf72da6b0018 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:19.559585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:19.559585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:19.559585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:19.559585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:19.559585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:19.559585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:19.559585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:19.559585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:19.559585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:19.559585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:19.559585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:19.559585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:28:19.559585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:28:19.559585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:28:19.560469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:28:19.560469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:19.560469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:19.560469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:19.560469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:19.560469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:19.560469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:19.560469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:19.560469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:19.560469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:19.560469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:19.560469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:19.560469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:19.560469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:19.560469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:19.560469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:19.560469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:19.561474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:28:19.561474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:28:19.561474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:28:19.561474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:28:19.561474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:28:19.561474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:28:19.561474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:19.561474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:19.565632 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 453/910] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:28:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:19.567035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204a50f0-e93c-4979-8cd3-92c74aa123c5 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:19.673379 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204a50f0-e93c-4979-8cd3-92c74aa123c5 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:19.701103 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:19.713088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7dae198-191d-4576-a19b-668b5f065330 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:19.713756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dae198-191d-4576-a19b-668b5f065330 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:19.717692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dae198-191d-4576-a19b-668b5f065330 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:19.717692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dae198-191d-4576-a19b-668b5f065330 None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:19.717692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7dae198-191d-4576-a19b-668b5f065330 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:19.717692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7dae198-191d-4576-a19b-668b5f065330 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:19.717692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7dae198-191d-4576-a19b-668b5f065330 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:19.729783 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 458/911] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:28:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:19.736246 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a479f5-0f1a-475f-80c4-bf72da6b0018 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:19.748839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:19.847416 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a479f5-0f1a-475f-80c4-bf72da6b0018 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:19.883562 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6ecafd-b9fb-439d-9acd-f3c4b9ff05e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:19.884508 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6ecafd-b9fb-439d-9acd-f3c4b9ff05e3 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:19.885008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6ecafd-b9fb-439d-9acd-f3c4b9ff05e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:19.885478 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6ecafd-b9fb-439d-9acd-f3c4b9ff05e3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:19.886383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6ecafd-b9fb-439d-9acd-f3c4b9ff05e3 None admin] RBAC: Adding query filter params (role.id=4b248d39ed014678bfc26b6f0acd7134, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=d40810f61ead4c3da414194b325da80a) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:19.886885 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6ecafd-b9fb-439d-9acd-f3c4b9ff05e3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:19.888594 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6ecafd-b9fb-439d-9acd-f3c4b9ff05e3 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:19.899032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 454/912] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:28:19 2022] GET /identity/v3/role_assignments?user.id=d40810f61ead4c3da414194b325da80a&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=4b248d39ed014678bfc26b6f0acd7134 => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:20.116328 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55c4979-15e9-44ee-81e2-13d5442bd82a None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:20.116631 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55c4979-15e9-44ee-81e2-13d5442bd82a None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:20.116809 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55c4979-15e9-44ee-81e2-13d5442bd82a None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:20.118361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 459/913] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:20.123845 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ab457e-0f91-4fe1-908c-19a39cdd0ee3 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:20.124135 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ab457e-0f91-4fe1-908c-19a39cdd0ee3 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:20.124412 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ab457e-0f91-4fe1-908c-19a39cdd0ee3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:20.143616 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a5ab457e-0f91-4fe1-908c-19a39cdd0ee3 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:20.144806 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a5ab457e-0f91-4fe1-908c-19a39cdd0ee3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:20.215996 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 455/914] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:20 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:20.510990 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55c4979-15e9-44ee-81e2-13d5442bd82a None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:20.617573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55c4979-15e9-44ee-81e2-13d5442bd82a None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:20.655830 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81546a1-f6fa-4104-91b2-410a8ed8e00f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:20.656971 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81546a1-f6fa-4104-91b2-410a8ed8e00f None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:20.657455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81546a1-f6fa-4104-91b2-410a8ed8e00f None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:20.657931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81546a1-f6fa-4104-91b2-410a8ed8e00f None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:20.658735 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81546a1-f6fa-4104-91b2-410a8ed8e00f None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:20.660414 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81546a1-f6fa-4104-91b2-410a8ed8e00f None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:20.670707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-c81546a1-f6fa-4104-91b2-410a8ed8e00f None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 16 03:28:20.670707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:20.670707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:20.670707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:20.670707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:20.670707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:20.670707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:20.670707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:20.670707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 16 03:28:20.670707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:20.670707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:20.670707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:20.670707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:20.670707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:20.670707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:20.670707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:20.672300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:20.672300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:20.672300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:20.672300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:20.672300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:20.672300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:20.672300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:20.672300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:20.672300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:20.672300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:20.672300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:20.672300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:20.672300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:20.672300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:20.672300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:20.672300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:20.672300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:20.673307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:20.673307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:20.673307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:20.673307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:20.673307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:20.673307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:20.673307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:20.673307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:20.673307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:20.673307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:20.673307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:20.673307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:20.673307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 16 03:28:20.673307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:20.676580 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 460/915] 10.222.0.49 () {56 vars in 1132 bytes} [Fri Sep 16 03:28:20 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:20.684449 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ab457e-0f91-4fe1-908c-19a39cdd0ee3 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:20.686641 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:20.775527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3552fc6d-2322-4192-86a3-fd11beb52f90 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:20.775722 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3552fc6d-2322-4192-86a3-fd11beb52f90 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:20.775986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3552fc6d-2322-4192-86a3-fd11beb52f90 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:20.777568 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 461/916] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:20.783674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3836e540-8de2-45b1-888d-87780f3ab019 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:20.783915 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3836e540-8de2-45b1-888d-87780f3ab019 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:20.784159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3836e540-8de2-45b1-888d-87780f3ab019 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:20.793808 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ab457e-0f91-4fe1-908c-19a39cdd0ee3 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:20.803250 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-3836e540-8de2-45b1-888d-87780f3ab019 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:20.804530 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3836e540-8de2-45b1-888d-87780f3ab019 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:20.835027 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83db1cb6-f7cc-4663-839e-58c4f7bdcd5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:20.835684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83db1cb6-f7cc-4663-839e-58c4f7bdcd5f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:20.835960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83db1cb6-f7cc-4663-839e-58c4f7bdcd5f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:20.836230 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83db1cb6-f7cc-4663-839e-58c4f7bdcd5f None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:20.836903 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83db1cb6-f7cc-4663-839e-58c4f7bdcd5f None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:20.837205 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83db1cb6-f7cc-4663-839e-58c4f7bdcd5f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:20.838604 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83db1cb6-f7cc-4663-839e-58c4f7bdcd5f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:20.844969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 456/917] 10.222.0.49 () {56 vars in 1126 bytes} [Fri Sep 16 03:28:20 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:20.850813 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83db1cb6-f7cc-4663-839e-58c4f7bdcd5f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:20.869378 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:20.898248 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 462/918] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:20 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:20.919644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:20.954469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83db1cb6-f7cc-4663-839e-58c4f7bdcd5f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:20.990195 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085ca655-bed9-4101-a54c-15611129e96d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:20.990804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085ca655-bed9-4101-a54c-15611129e96d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:20.991120 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085ca655-bed9-4101-a54c-15611129e96d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:20.991398 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085ca655-bed9-4101-a54c-15611129e96d None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:20.992026 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085ca655-bed9-4101-a54c-15611129e96d None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:20.992328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085ca655-bed9-4101-a54c-15611129e96d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:20.993684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085ca655-bed9-4101-a54c-15611129e96d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:20.999218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 457/919] 10.222.0.49 () {56 vars in 1126 bytes} [Fri Sep 16 03:28:20 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:21.006551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3836e540-8de2-45b1-888d-87780f3ab019 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:21.119087 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3836e540-8de2-45b1-888d-87780f3ab019 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:21.132876 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5f0430-0a7f-449a-a34a-b0b1a5ef1600 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:21.133166 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5f0430-0a7f-449a-a34a-b0b1a5ef1600 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:21.133443 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5f0430-0a7f-449a-a34a-b0b1a5ef1600 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:21.135063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 458/920] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:21.139896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7925d3-8be3-4a66-b49f-5038dabf3617 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:21.140172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7925d3-8be3-4a66-b49f-5038dabf3617 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:21.141025 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7925d3-8be3-4a66-b49f-5038dabf3617 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:21.157661 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d4ebd2-a573-42a5-aa85-0f8820e5e7c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:21.158580 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d4ebd2-a573-42a5-aa85-0f8820e5e7c9 None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:21.159143 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d4ebd2-a573-42a5-aa85-0f8820e5e7c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:21.159599 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d4ebd2-a573-42a5-aa85-0f8820e5e7c9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:21.160443 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d4ebd2-a573-42a5-aa85-0f8820e5e7c9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:21.160751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ac7925d3-8be3-4a66-b49f-5038dabf3617 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:21.161945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ac7925d3-8be3-4a66-b49f-5038dabf3617 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:21.162422 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d4ebd2-a573-42a5-aa85-0f8820e5e7c9 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:21.232586 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 459/921] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:21 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 03:28:21.235047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7925d3-8be3-4a66-b49f-5038dabf3617 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:21.252238 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 463/922] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:28:21 2022] POST /identity/v3/endpoints => generated 347 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:28:21.319740 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:21.371338 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7925d3-8be3-4a66-b49f-5038dabf3617 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:21.431078 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230556b6-eeb5-4f8f-9483-49ed65778abe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:21.431812 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230556b6-eeb5-4f8f-9483-49ed65778abe None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:21.432097 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230556b6-eeb5-4f8f-9483-49ed65778abe None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:21.432372 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230556b6-eeb5-4f8f-9483-49ed65778abe None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:21.432978 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230556b6-eeb5-4f8f-9483-49ed65778abe None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:21.434394 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230556b6-eeb5-4f8f-9483-49ed65778abe None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:21.446898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-230556b6-eeb5-4f8f-9483-49ed65778abe None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 03:28:21.446898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:21.446898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:21.446898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:21.446898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:21.446898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:21.446898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:21.446898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:21.446898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 03:28:21.446898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:21.446898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:21.446898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:21.446898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:21.446898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:21.446898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:21.446898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:21.446898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:21.447966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:21.447966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:21.447966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:21.447966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:21.447966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:21.447966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:21.447966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:21.447966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:21.447966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:21.447966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:21.447966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:21.447966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:21.447966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:21.447966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:21.447966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:21.447966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:21.448823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:21.448823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:21.448823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:21.448823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:21.448823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:21.448823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:21.448823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:21.448823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:21.448823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:21.448823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:21.448823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:21.448823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:21.448823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 03:28:21.448823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:21.452968 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 460/923] 10.222.0.49 () {56 vars in 1126 bytes} [Fri Sep 16 03:28:21 2022] GET /identity/v3/services/object-store => generated 93 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:21.454743 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d4ebd2-a573-42a5-aa85-0f8820e5e7c9 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:21.491040 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:21.506136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230556b6-eeb5-4f8f-9483-49ed65778abe None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:21.568014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d4ebd2-a573-42a5-aa85-0f8820e5e7c9 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:21.614299 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61d7c1d-dcee-46ad-ad5f-dbddb5e9e708 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:21.615342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61d7c1d-dcee-46ad-ad5f-dbddb5e9e708 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:21.615880 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61d7c1d-dcee-46ad-ad5f-dbddb5e9e708 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:21.616159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61d7c1d-dcee-46ad-ad5f-dbddb5e9e708 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:21.616791 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61d7c1d-dcee-46ad-ad5f-dbddb5e9e708 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:21.617342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61d7c1d-dcee-46ad-ad5f-dbddb5e9e708 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:21.619547 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61d7c1d-dcee-46ad-ad5f-dbddb5e9e708 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:21.624229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230556b6-eeb5-4f8f-9483-49ed65778abe None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:21.626404 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 464/924] 10.222.0.49 () {56 vars in 1122 bytes} [Fri Sep 16 03:28:21 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:21.631975 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61d7c1d-dcee-46ad-ad5f-dbddb5e9e708 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:21.650521 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:21.666672 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd579382-13ef-4b2e-bb10-0d4365b1894c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:21.667357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd579382-13ef-4b2e-bb10-0d4365b1894c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:21.667692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd579382-13ef-4b2e-bb10-0d4365b1894c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:21.667975 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd579382-13ef-4b2e-bb10-0d4365b1894c None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:21.668562 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd579382-13ef-4b2e-bb10-0d4365b1894c None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:21.670152 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd579382-13ef-4b2e-bb10-0d4365b1894c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:21.679958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-dd579382-13ef-4b2e-bb10-0d4365b1894c None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 16 03:28:21.679958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:21.679958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:21.679958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:21.679958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:21.679958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:21.679958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:21.679958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:21.679958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 16 03:28:21.679958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:21.679958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:21.679958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:21.679958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:21.679958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:21.679958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:21.679958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:21.679958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:21.681301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:21.681301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:21.681301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:21.681301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:21.681301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:21.681301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:21.681301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:21.681301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:21.681301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:21.681301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:21.681301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:21.681301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:21.681301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:21.681301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:21.681301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:21.681301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:21.682361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:21.682361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:21.682361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:21.682361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:21.682361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:21.682361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:21.682361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:21.682361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:21.682361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:21.682361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:21.682361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:21.682361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:21.682361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 16 03:28:21.682361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:21.686040 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 461/925] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:28:21 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:21.687987 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd579382-13ef-4b2e-bb10-0d4365b1894c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:21.766457 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:21.772033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61d7c1d-dcee-46ad-ad5f-dbddb5e9e708 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:21.816120 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd579382-13ef-4b2e-bb10-0d4365b1894c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:21.818404 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed6bd58-0ea9-4bb2-a567-4704db792c09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:21.819619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed6bd58-0ea9-4bb2-a567-4704db792c09 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:21.819907 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed6bd58-0ea9-4bb2-a567-4704db792c09 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:21.820185 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed6bd58-0ea9-4bb2-a567-4704db792c09 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:21.822960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed6bd58-0ea9-4bb2-a567-4704db792c09 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:21.823514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed6bd58-0ea9-4bb2-a567-4704db792c09 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:21.825106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed6bd58-0ea9-4bb2-a567-4704db792c09 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:21.832601 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 465/926] 10.222.0.49 () {56 vars in 1122 bytes} [Fri Sep 16 03:28:21 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:21.839143 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed6bd58-0ea9-4bb2-a567-4704db792c09 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:21.861581 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c69182-455d-44b4-849d-6ce2a9b1a313 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:21.862045 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c69182-455d-44b4-849d-6ce2a9b1a313 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:21.862334 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c69182-455d-44b4-849d-6ce2a9b1a313 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:21.862611 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c69182-455d-44b4-849d-6ce2a9b1a313 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:21.863278 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c69182-455d-44b4-849d-6ce2a9b1a313 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:21.863575 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c69182-455d-44b4-849d-6ce2a9b1a313 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:21.865021 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c69182-455d-44b4-849d-6ce2a9b1a313 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:21.870239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 462/927] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:28:21 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:21.875559 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c69182-455d-44b4-849d-6ce2a9b1a313 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:21.939266 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:21.948835 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed6bd58-0ea9-4bb2-a567-4704db792c09 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:21.983796 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9078ac7b-e572-4284-9540-ff48cba31db2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:21.985401 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9078ac7b-e572-4284-9540-ff48cba31db2 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:21.985896 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9078ac7b-e572-4284-9540-ff48cba31db2 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:21.986366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9078ac7b-e572-4284-9540-ff48cba31db2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:21.987305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9078ac7b-e572-4284-9540-ff48cba31db2 None admin] RBAC: Adding query filter params (interface=public, service_id=d0aefe94b2db42b7a4c4ea33267a862f, region_id=RegionOne) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:21.987841 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9078ac7b-e572-4284-9540-ff48cba31db2 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:21.989493 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9078ac7b-e572-4284-9540-ff48cba31db2 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:21.993948 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c69182-455d-44b4-849d-6ce2a9b1a313 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:21.995646 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 466/928] 10.222.0.49 () {56 vars in 1251 bytes} [Fri Sep 16 03:28:21 2022] GET /identity/v3/endpoints?service_id=d0aefe94b2db42b7a4c4ea33267a862f&interface=public®ion_id=RegionOne => generated 194 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:22.001877 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9078ac7b-e572-4284-9540-ff48cba31db2 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:22.037069 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:22.041188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d3c930-cc3d-4160-b8b8-cef7c57c0b5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:22.042044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d3c930-cc3d-4160-b8b8-cef7c57c0b5a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:22.042523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d3c930-cc3d-4160-b8b8-cef7c57c0b5a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:22.043080 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d3c930-cc3d-4160-b8b8-cef7c57c0b5a None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:22.044094 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d3c930-cc3d-4160-b8b8-cef7c57c0b5a None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:22.049502 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d3c930-cc3d-4160-b8b8-cef7c57c0b5a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:22.049502 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d3c930-cc3d-4160-b8b8-cef7c57c0b5a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:22.052562 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 463/929] 10.222.0.49 () {56 vars in 1114 bytes} [Fri Sep 16 03:28:21 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:22.060016 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d3c930-cc3d-4160-b8b8-cef7c57c0b5a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:22.120917 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9078ac7b-e572-4284-9540-ff48cba31db2 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:22.159511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2108c40-6e17-4009-a37b-f93617eb60d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:22.160165 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2108c40-6e17-4009-a37b-f93617eb60d7 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:22.160453 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2108c40-6e17-4009-a37b-f93617eb60d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:22.160727 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2108c40-6e17-4009-a37b-f93617eb60d7 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:22.161334 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2108c40-6e17-4009-a37b-f93617eb60d7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:22.162712 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2108c40-6e17-4009-a37b-f93617eb60d7 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:22.169633 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 467/930] 10.222.0.49 () {56 vars in 1087 bytes} [Fri Sep 16 03:28:22 2022] GET /identity/v3/services => generated 1497 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:28:22.185039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba088c23-c6f8-46b4-8854-16ced34e0f9e None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:22.185383 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba088c23-c6f8-46b4-8854-16ced34e0f9e None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:22.185655 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba088c23-c6f8-46b4-8854-16ced34e0f9e None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:22.188088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 468/931] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:28:22.189167 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d3c930-cc3d-4160-b8b8-cef7c57c0b5a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:22.196624 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d80112-d2b3-4ddd-a130-7b70c9a5b5aa None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:22.196979 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d80112-d2b3-4ddd-a130-7b70c9a5b5aa None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:22.197311 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d80112-d2b3-4ddd-a130-7b70c9a5b5aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:22.209066 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:22.218232 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-c9d80112-d2b3-4ddd-a130-7b70c9a5b5aa None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:22.220360 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c9d80112-d2b3-4ddd-a130-7b70c9a5b5aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:22.233204 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e826bad4-1273-4293-8f99-688f97e8b5d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:22.233790 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e826bad4-1273-4293-8f99-688f97e8b5d4 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:22.234216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e826bad4-1273-4293-8f99-688f97e8b5d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:22.234611 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e826bad4-1273-4293-8f99-688f97e8b5d4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:22.235546 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e826bad4-1273-4293-8f99-688f97e8b5d4 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:22.237200 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e826bad4-1273-4293-8f99-688f97e8b5d4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:22.305659 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 469/932] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:22 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 03:28:22.347180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 464/933] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:28:22 2022] POST /identity/v3/endpoints => generated 346 bytes in 287 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:28:22.586400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d80112-d2b3-4ddd-a130-7b70c9a5b5aa None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:22.608740 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:22.689500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d80112-d2b3-4ddd-a130-7b70c9a5b5aa None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:22.731302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f570901-f738-44ba-8fed-a66841586f35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:22.732356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f570901-f738-44ba-8fed-a66841586f35 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:22.732847 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f570901-f738-44ba-8fed-a66841586f35 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:22.733334 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f570901-f738-44ba-8fed-a66841586f35 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:22.745038 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f570901-f738-44ba-8fed-a66841586f35 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:28:22.750469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-4f570901-f738-44ba-8fed-a66841586f35 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 03:28:22.750469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:22.750469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:22.750469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:22.750469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:22.750469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:22.750469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:22.750469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:22.750469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:22.750469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:22.750469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:22.750469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:22.750469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:28:22.750469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:28:22.750469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:28:22.750469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:28:22.751489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:28:22.751489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:28:22.751489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:28:22.751489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:28:22.751489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:22.751489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:22.751489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:22.751489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:22.751489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:22.751489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:22.751489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:22.751489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:22.751489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:22.751489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:22.751489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:22.751489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:22.752346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:22.752346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:22.752346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:22.752346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:22.752346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:28:22.752346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:28:22.752346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:28:22.752346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:28:22.752346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:28:22.752346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:28:22.752346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 03:28:22.752346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:22.755032 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 470/934] 10.222.0.49 () {56 vars in 1120 bytes} [Fri Sep 16 03:28:22 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:22.760736 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e826bad4-1273-4293-8f99-688f97e8b5d4 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:22.866375 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e826bad4-1273-4293-8f99-688f97e8b5d4 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:22.904249 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c8d702-9660-4a12-9865-d8eba3c843c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:22.904858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c8d702-9660-4a12-9865-d8eba3c843c1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:22.905136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c8d702-9660-4a12-9865-d8eba3c843c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:22.905407 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c8d702-9660-4a12-9865-d8eba3c843c1 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:22.906360 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c8d702-9660-4a12-9865-d8eba3c843c1 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:22.906662 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c8d702-9660-4a12-9865-d8eba3c843c1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:22.908043 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c8d702-9660-4a12-9865-d8eba3c843c1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:22.920946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 465/935] 10.222.0.49 () {56 vars in 1115 bytes} [Fri Sep 16 03:28:22 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:22.922868 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f570901-f738-44ba-8fed-a66841586f35 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:23.008171 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:23.032800 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f570901-f738-44ba-8fed-a66841586f35 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:23.068807 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20be0c0e-2bd9-47c3-8052-3822958f6689 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:23.069443 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20be0c0e-2bd9-47c3-8052-3822958f6689 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:23.069746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20be0c0e-2bd9-47c3-8052-3822958f6689 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:23.070017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20be0c0e-2bd9-47c3-8052-3822958f6689 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:23.080831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20be0c0e-2bd9-47c3-8052-3822958f6689 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:23.082287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20be0c0e-2bd9-47c3-8052-3822958f6689 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:23.097750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-20be0c0e-2bd9-47c3-8052-3822958f6689 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:23.097750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:23.097750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:23.097750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:23.097750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:23.097750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:23.097750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:23.097750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:23.097750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:23.097750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:23.097750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:23.097750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:23.097750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:23.097750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:23.097750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:23.097750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:23.098744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:23.098744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:23.098744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:23.098744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:23.098744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:23.098744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:23.098744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:23.098744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:23.098744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:23.098744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:23.098744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:23.098744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:23.098744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:23.098744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:23.098744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:23.098744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:23.098744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:23.099834 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:23.099834 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:23.099834 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:23.099834 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 471/936] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:23.106241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c8d702-9660-4a12-9865-d8eba3c843c1 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:23.142210 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:23.215345 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c8d702-9660-4a12-9865-d8eba3c843c1 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:23.250304 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73a7afa-2e83-428f-aa51-b6d69e143342 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:23.251460 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73a7afa-2e83-428f-aa51-b6d69e143342 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:23.251953 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73a7afa-2e83-428f-aa51-b6d69e143342 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:23.252433 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73a7afa-2e83-428f-aa51-b6d69e143342 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:23.253318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73a7afa-2e83-428f-aa51-b6d69e143342 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:23.253812 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73a7afa-2e83-428f-aa51-b6d69e143342 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:23.255404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73a7afa-2e83-428f-aa51-b6d69e143342 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:23.273004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 466/937] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:23.280044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20be0c0e-2bd9-47c3-8052-3822958f6689 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:23.394024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20be0c0e-2bd9-47c3-8052-3822958f6689 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:23.422475 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:23.431139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072ddf3c-e96f-468f-9290-7706b3b666d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:23.436838 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072ddf3c-e96f-468f-9290-7706b3b666d7 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:23.436838 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072ddf3c-e96f-468f-9290-7706b3b666d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:23.436838 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072ddf3c-e96f-468f-9290-7706b3b666d7 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:23.440758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072ddf3c-e96f-468f-9290-7706b3b666d7 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:23.442401 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072ddf3c-e96f-468f-9290-7706b3b666d7 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:23.455330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-072ddf3c-e96f-468f-9290-7706b3b666d7 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 03:28:23.455330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:23.455330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:23.455330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:23.455330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:23.455330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:23.455330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:23.455330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:23.455330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:23.455330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:23.455330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:23.455330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:23.455330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:28:23.455330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:28:23.455330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:28:23.455330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:28:23.456321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:23.456321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:23.456321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:28:23.456321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:23.456321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:28:23.456321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:23.456321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:23.456321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:23.456321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:23.456321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:23.456321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:23.456321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:23.456321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:23.456321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:23.456321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:23.456321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:23.456321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:23.456321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:23.457328 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:23.457328 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:23.457328 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:28:23.457328 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:28:23.457328 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:28:23.457328 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:28:23.457328 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:28:23.457328 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:28:23.457328 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 03:28:23.457328 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:23.459543 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 472/938] 10.222.0.49 () {56 vars in 1117 bytes} [Fri Sep 16 03:28:23 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:23.465667 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73a7afa-2e83-428f-aa51-b6d69e143342 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:23.559981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:23.571083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73a7afa-2e83-428f-aa51-b6d69e143342 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:23.608423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6da7138-ba38-4f03-9bb6-77ce3c29c193 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:23.609572 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6da7138-ba38-4f03-9bb6-77ce3c29c193 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:23.609873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6da7138-ba38-4f03-9bb6-77ce3c29c193 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:23.610138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6da7138-ba38-4f03-9bb6-77ce3c29c193 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:23.610862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6da7138-ba38-4f03-9bb6-77ce3c29c193 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:23.611192 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6da7138-ba38-4f03-9bb6-77ce3c29c193 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:23.612606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6da7138-ba38-4f03-9bb6-77ce3c29c193 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:23.635028 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 467/939] 10.222.0.49 () {56 vars in 1149 bytes} [Fri Sep 16 03:28:23 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:23.636711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072ddf3c-e96f-468f-9290-7706b3b666d7 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:23.691868 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9be5b8-3cbc-41b7-856f-f7e4b3c3df41 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:23.692058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9be5b8-3cbc-41b7-856f-f7e4b3c3df41 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:23.692344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9be5b8-3cbc-41b7-856f-f7e4b3c3df41 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:23.694017 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 468/940] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:28:23.698713 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa36764e-7136-4dfa-b873-ee4eb1d823ae None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:23.699029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa36764e-7136-4dfa-b873-ee4eb1d823ae None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:23.699306 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa36764e-7136-4dfa-b873-ee4eb1d823ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:23.719269 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-fa36764e-7136-4dfa-b873-ee4eb1d823ae None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:23.720450 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fa36764e-7136-4dfa-b873-ee4eb1d823ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:23.748874 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072ddf3c-e96f-468f-9290-7706b3b666d7 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:23.786441 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81f78b1-2b64-4726-a414-77f8ca7325d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:23.787560 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81f78b1-2b64-4726-a414-77f8ca7325d4 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:23.788166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81f78b1-2b64-4726-a414-77f8ca7325d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:23.788665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81f78b1-2b64-4726-a414-77f8ca7325d4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:23.798036 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 469/941] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:23 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 03:28:23.802103 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81f78b1-2b64-4726-a414-77f8ca7325d4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:23.803643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81f78b1-2b64-4726-a414-77f8ca7325d4 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:23.818645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-b81f78b1-2b64-4726-a414-77f8ca7325d4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:23.818645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:23.818645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:23.818645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:23.818645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:23.818645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:23.818645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:23.818645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:23.818645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:23.818645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:23.818645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:23.818645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:23.818645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:23.818645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:23.818645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:23.818645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:23.819639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:23.819639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:23.819639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:23.819639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:23.819639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:23.819639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:23.819639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:23.819639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:23.819639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:23.819639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:23.819639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:23.819639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:23.819639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:23.819639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:23.819639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:23.819639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:23.819639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:23.820551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:23.820551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:23.820551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:23.824429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 473/942] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:23.825791 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa36764e-7136-4dfa-b873-ee4eb1d823ae None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:23.908205 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:23.927089 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa36764e-7136-4dfa-b873-ee4eb1d823ae None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:23.967603 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e45c1b9-1d74-40cb-939a-7089048f419a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:23.968693 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e45c1b9-1d74-40cb-939a-7089048f419a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:23.969204 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e45c1b9-1d74-40cb-939a-7089048f419a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:23.969768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e45c1b9-1d74-40cb-939a-7089048f419a None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:23.970626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e45c1b9-1d74-40cb-939a-7089048f419a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:23.971166 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e45c1b9-1d74-40cb-939a-7089048f419a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:23.972745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e45c1b9-1d74-40cb-939a-7089048f419a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:23.985235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 470/943] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:23.993702 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81f78b1-2b64-4726-a414-77f8ca7325d4 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:24.035939 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:24.101311 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81f78b1-2b64-4726-a414-77f8ca7325d4 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:24.106179 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e45c1b9-1d74-40cb-939a-7089048f419a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:24.146201 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3633a4-fe30-4b4e-9d1d-4935b74b2f32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:24.146843 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3633a4-fe30-4b4e-9d1d-4935b74b2f32 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:24.147156 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3633a4-fe30-4b4e-9d1d-4935b74b2f32 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:24.147537 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3633a4-fe30-4b4e-9d1d-4935b74b2f32 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:24.156814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3633a4-fe30-4b4e-9d1d-4935b74b2f32 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:24.158700 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3633a4-fe30-4b4e-9d1d-4935b74b2f32 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:24.175695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-ef3633a4-fe30-4b4e-9d1d-4935b74b2f32 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:24.175695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:24.175695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:24.175695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:24.175695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:24.175695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:24.175695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:24.175695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:24.175695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:24.175695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:24.175695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:24.175695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:24.175695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:28:24.175695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:28:24.175695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:28:24.176663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:28:24.176663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:24.176663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:24.176663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:24.176663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:24.176663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:24.176663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:24.176663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:24.176663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:24.176663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:24.176663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:24.176663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:24.176663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:24.176663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:24.176663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:24.176663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:24.176663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:24.177603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:28:24.177603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:28:24.177603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:28:24.177603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:28:24.177603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:28:24.177603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:28:24.177603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:24.177603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:24.178322 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 474/944] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:28:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:24.184189 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3633a4-fe30-4b4e-9d1d-4935b74b2f32 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:24.220122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e45c1b9-1d74-40cb-939a-7089048f419a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:24.261544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb12b18-a21f-4dc7-b34c-3295068d5b80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:24.262539 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb12b18-a21f-4dc7-b34c-3295068d5b80 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:24.263083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb12b18-a21f-4dc7-b34c-3295068d5b80 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:24.263571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb12b18-a21f-4dc7-b34c-3295068d5b80 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:24.264358 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb12b18-a21f-4dc7-b34c-3295068d5b80 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:24.266185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb12b18-a21f-4dc7-b34c-3295068d5b80 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:24.276041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-5bb12b18-a21f-4dc7-b34c-3295068d5b80 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 16 03:28:24.276041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:24.276041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:24.276041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:24.276041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:24.276041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:24.276041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:24.276041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:24.276041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 16 03:28:24.276041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:24.276041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:24.276041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:24.276041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:24.276041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:24.276041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:24.276041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:24.276041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:24.277255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:24.277255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:24.277255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:24.277255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:24.277255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:24.277255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:24.277255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:24.277255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:24.277255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:24.277255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:24.277255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:24.277255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:24.277255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:24.277255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:24.277255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:24.277255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:24.278144 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:24.278144 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:24.278144 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:24.278144 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:24.278144 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:24.278144 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:24.278144 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:24.278144 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:24.278144 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:24.278144 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:24.278144 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:24.278144 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:24.278144 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 16 03:28:24.278144 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:24.286029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 471/945] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:28:24 2022] GET /identity/v3/services/compute => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:24.287522 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb12b18-a21f-4dc7-b34c-3295068d5b80 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:24.292329 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:24.302847 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:24.307145 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3633a4-fe30-4b4e-9d1d-4935b74b2f32 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:24.353438 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734ac8fb-5cb0-45a7-8dd1-d935fa397f46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:24.354379 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734ac8fb-5cb0-45a7-8dd1-d935fa397f46 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:24.354884 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734ac8fb-5cb0-45a7-8dd1-d935fa397f46 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:24.355417 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734ac8fb-5cb0-45a7-8dd1-d935fa397f46 None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:24.356406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734ac8fb-5cb0-45a7-8dd1-d935fa397f46 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:24.356906 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734ac8fb-5cb0-45a7-8dd1-d935fa397f46 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:24.358532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734ac8fb-5cb0-45a7-8dd1-d935fa397f46 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:24.379365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 475/946] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:28:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:24.381184 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734ac8fb-5cb0-45a7-8dd1-d935fa397f46 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:24.414818 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb12b18-a21f-4dc7-b34c-3295068d5b80 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:24.454067 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d63f0c-c900-4766-8c14-bb27ef4184d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:24.454697 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d63f0c-c900-4766-8c14-bb27ef4184d6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:24.454988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d63f0c-c900-4766-8c14-bb27ef4184d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:24.455261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d63f0c-c900-4766-8c14-bb27ef4184d6 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:24.455887 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d63f0c-c900-4766-8c14-bb27ef4184d6 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:24.456186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d63f0c-c900-4766-8c14-bb27ef4184d6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:24.457543 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d63f0c-c900-4766-8c14-bb27ef4184d6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:24.468086 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 472/947] 10.222.0.49 () {56 vars in 1112 bytes} [Fri Sep 16 03:28:24 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:24.472182 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d63f0c-c900-4766-8c14-bb27ef4184d6 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:24.508192 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734ac8fb-5cb0-45a7-8dd1-d935fa397f46 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:24.546408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:24.553177 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942f8a7f-0235-4e2a-8ec8-72b8daac59c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:24.553838 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942f8a7f-0235-4e2a-8ec8-72b8daac59c2 None admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:24.554122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942f8a7f-0235-4e2a-8ec8-72b8daac59c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:24.554406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942f8a7f-0235-4e2a-8ec8-72b8daac59c2 None admin] PATH_INFO: `/v3/projects/0bb7350474f349d6aee197ff35536167/users/d40810f61ead4c3da414194b325da80a/roles/4b248d39ed014678bfc26b6f0acd7134` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:24.556566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942f8a7f-0235-4e2a-8ec8-72b8daac59c2 None admin] RBAC: Authorizing `identity:create_grant(project_id=0bb7350474f349d6aee197ff35536167, user_id=d40810f61ead4c3da414194b325da80a, role_id=4b248d39ed014678bfc26b6f0acd7134)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:24.558011 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942f8a7f-0235-4e2a-8ec8-72b8daac59c2 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:24.582630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d63f0c-c900-4766-8c14-bb27ef4184d6 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:24.616096 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18bccad-ec93-4e9a-bd83-52b3c18e33f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:24.616705 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18bccad-ec93-4e9a-bd83-52b3c18e33f7 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:24.616986 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18bccad-ec93-4e9a-bd83-52b3c18e33f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:24.617260 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18bccad-ec93-4e9a-bd83-52b3c18e33f7 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:24.617881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18bccad-ec93-4e9a-bd83-52b3c18e33f7 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:24.618182 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18bccad-ec93-4e9a-bd83-52b3c18e33f7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:24.619575 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18bccad-ec93-4e9a-bd83-52b3c18e33f7 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:24.623841 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 473/948] 10.222.0.49 () {56 vars in 1112 bytes} [Fri Sep 16 03:28:24 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:24.625681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d724da5b-fb90-4eab-a05b-b2c67ed0cec8 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:24.625973 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d724da5b-fb90-4eab-a05b-b2c67ed0cec8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:24.626247 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d724da5b-fb90-4eab-a05b-b2c67ed0cec8 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:24.627666 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 474/949] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:24.628605 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c873cfea-76f8-4837-8e9c-b1bfa63e82e4 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:24.628888 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c873cfea-76f8-4837-8e9c-b1bfa63e82e4 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:24.629161 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c873cfea-76f8-4837-8e9c-b1bfa63e82e4 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:24.630585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 475/950] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:24 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:24.633478 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9d4ccc-4859-44cb-b457-3da43d4cc053 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:24.633767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9d4ccc-4859-44cb-b457-3da43d4cc053 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:24.634044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9d4ccc-4859-44cb-b457-3da43d4cc053 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:24.643677 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 476/951] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:28:24 2022] PUT /identity/v3/projects/0bb7350474f349d6aee197ff35536167/users/d40810f61ead4c3da414194b325da80a/roles/4b248d39ed014678bfc26b6f0acd7134 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:28:24.644889 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c7e53e-3f50-4eb6-b544-c4ae194ba7e2 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:24.645145 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c7e53e-3f50-4eb6-b544-c4ae194ba7e2 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:24.645419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c7e53e-3f50-4eb6-b544-c4ae194ba7e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:24.663396 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7f9d4ccc-4859-44cb-b457-3da43d4cc053 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:24.665199 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7f9d4ccc-4859-44cb-b457-3da43d4cc053 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:24.666955 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-92c7e53e-3f50-4eb6-b544-c4ae194ba7e2 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:24.668268 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-92c7e53e-3f50-4eb6-b544-c4ae194ba7e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:24.691532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:24.760683 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 476/952] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:24 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 03:28:24.761193 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 477/953] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:24 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:25.029937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c7e53e-3f50-4eb6-b544-c4ae194ba7e2 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:25.035406 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9d4ccc-4859-44cb-b457-3da43d4cc053 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:25.098803 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:25.133653 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c7e53e-3f50-4eb6-b544-c4ae194ba7e2 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:25.137866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9d4ccc-4859-44cb-b457-3da43d4cc053 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:25.169354 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff32d7b-aee3-4722-9247-379e9975fb4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:25.170076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff32d7b-aee3-4722-9247-379e9975fb4f None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:25.170361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff32d7b-aee3-4722-9247-379e9975fb4f None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:25.170639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff32d7b-aee3-4722-9247-379e9975fb4f None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:25.171339 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff32d7b-aee3-4722-9247-379e9975fb4f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:25.173512 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff32d7b-aee3-4722-9247-379e9975fb4f None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:25.174061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becd8ce9-3477-465c-a6c9-66b3ef6d1416 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:25.174704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becd8ce9-3477-465c-a6c9-66b3ef6d1416 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:25.175009 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becd8ce9-3477-465c-a6c9-66b3ef6d1416 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:25.175280 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becd8ce9-3477-465c-a6c9-66b3ef6d1416 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:25.175857 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-becd8ce9-3477-465c-a6c9-66b3ef6d1416 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:25.176151 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: WARNING keystone.server.flask.common [None req-6ff32d7b-aee3-4722-9247-379e9975fb4f None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 16 03:28:25.177204 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-becd8ce9-3477-465c-a6c9-66b3ef6d1416 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:25.186323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-becd8ce9-3477-465c-a6c9-66b3ef6d1416 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 03:28:25.186323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:25.186323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:25.186323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:25.186323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:25.186323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:25.186323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:25.186323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:25.186323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 03:28:25.186323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:25.186323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:25.186323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:25.186323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:25.186323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:25.186323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:25.186323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:25.186323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:25.187435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:25.187435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:25.187435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:25.187435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:25.187435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:25.187435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:25.187435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:25.187435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:25.187435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:25.187435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:25.187435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:25.187435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:25.187435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:25.187435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:25.187435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:25.187435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:25.188294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:25.188294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:25.188294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:25.188294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:25.188294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:25.188294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:25.188294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:25.188294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:25.188294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:25.188294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:25.188294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:25.188294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:25.188294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 03:28:25.188294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:25.192760 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 477/954] 10.222.0.49 () {56 vars in 1126 bytes} [Fri Sep 16 03:28:25 2022] GET /identity/v3/services/object-store => generated 93 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:25.194315 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becd8ce9-3477-465c-a6c9-66b3ef6d1416 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:25.222091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.sql.core [None req-6ff32d7b-aee3-4722-9247-379e9975fb4f None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 16 03:28:25.222091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 16 03:28:25.222091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [parameters: {'id': 'ebd9ee2982a94a5481fca1ebd87b12dc', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 16 03:28:25.222091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81761) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 16 03:28:25.230901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-6ff32d7b-aee3-4722-9247-379e9975fb4f None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id ebd9ee2982a94a5481fca1ebd87b12dc.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id ebd9ee2982a94a5481fca1ebd87b12dc. Sep 16 03:28:25.230901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:25.230901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 16 03:28:25.230901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 16 03:28:25.230901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 16 03:28:25.230901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 16 03:28:25.230901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 16 03:28:25.230901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application result = self._query(query) Sep 16 03:28:25.230901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 16 03:28:25.230901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application conn.query(q) Sep 16 03:28:25.230901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 16 03:28:25.230901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 16 03:28:25.230901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 16 03:28:25.231764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application result.read() Sep 16 03:28:25.231764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 16 03:28:25.231764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 16 03:28:25.231764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 16 03:28:25.231764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 16 03:28:25.231764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 16 03:28:25.231764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 16 03:28:25.231764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 16 03:28:25.231764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 16 03:28:25.231764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 16 03:28:25.231764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:25.231764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 16 03:28:25.231764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:25.231764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:25.231764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 16 03:28:25.231764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 16 03:28:25.231764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 16 03:28:25.231764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 16 03:28:25.232867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 16 03:28:25.232867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application next(self.gen) Sep 16 03:28:25.232867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 16 03:28:25.232867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application yield resource Sep 16 03:28:25.232867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 16 03:28:25.232867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application next(self.gen) Sep 16 03:28:25.232867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 16 03:28:25.232867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self.session.rollback() Sep 16 03:28:25.232867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 16 03:28:25.232867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self.force_reraise() Sep 16 03:28:25.232867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 16 03:28:25.232867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise self.value Sep 16 03:28:25.232867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 16 03:28:25.232867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 16 03:28:25.232867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 16 03:28:25.232867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application session.commit() Sep 16 03:28:25.232867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 16 03:28:25.233788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 16 03:28:25.233788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 16 03:28:25.233788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._prepare_impl() Sep 16 03:28:25.233788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 16 03:28:25.233788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self.session.flush() Sep 16 03:28:25.233788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 16 03:28:25.233788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._flush(objects) Sep 16 03:28:25.233788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 16 03:28:25.233788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 16 03:28:25.233788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 16 03:28:25.233788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application compat.raise_( Sep 16 03:28:25.233788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 16 03:28:25.233788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception Sep 16 03:28:25.233788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 16 03:28:25.233788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application flush_context.execute() Sep 16 03:28:25.233788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 16 03:28:25.233788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rec.execute(self) Sep 16 03:28:25.234777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 16 03:28:25.234777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 16 03:28:25.234777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 16 03:28:25.234777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 16 03:28:25.234777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 16 03:28:25.234777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 16 03:28:25.234777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 16 03:28:25.234777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 16 03:28:25.234777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 16 03:28:25.234777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 16 03:28:25.234777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 16 03:28:25.234777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 16 03:28:25.234777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 16 03:28:25.234777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 16 03:28:25.234777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 16 03:28:25.237154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 16 03:28:25.237154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 16 03:28:25.237154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception Sep 16 03:28:25.237154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 16 03:28:25.237154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 16 03:28:25.237154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 16 03:28:25.237154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 16 03:28:25.237154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 16 03:28:25.237154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application result = self._query(query) Sep 16 03:28:25.237154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 16 03:28:25.237154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application conn.query(q) Sep 16 03:28:25.237154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 16 03:28:25.237154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 16 03:28:25.237154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 16 03:28:25.237154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application result.read() Sep 16 03:28:25.237154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 16 03:28:25.237154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 16 03:28:25.238704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 16 03:28:25.238704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 16 03:28:25.238704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 16 03:28:25.238704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 16 03:28:25.238704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 16 03:28:25.238704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 16 03:28:25.238704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 16 03:28:25.238704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 16 03:28:25.238704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [parameters: {'id': 'ebd9ee2982a94a5481fca1ebd87b12dc', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 16 03:28:25.238704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 16 03:28:25.238704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:25.238704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:25.238704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:25.238704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:25.238704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 16 03:28:25.239832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 16 03:28:25.239832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 16 03:28:25.239832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 16 03:28:25.239832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 16 03:28:25.239832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:25.239832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:25.239832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:25.239832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:25.239832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:25.239832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:25.239832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:25.239832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:25.239832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:25.239832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:25.239832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:25.239832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:25.239832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:25.241014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:25.241014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 16 03:28:25.241014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 16 03:28:25.241014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:25.241014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:25.241014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 16 03:28:25.241014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 16 03:28:25.241014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id ebd9ee2982a94a5481fca1ebd87b12dc. Sep 16 03:28:25.241014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:25.248636 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 478/955] 10.222.0.49 () {60 vars in 1140 bytes} [Fri Sep 16 03:28:25 2022] POST /identity/v3/projects => generated 262 bytes in 214 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:25.250198 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff32d7b-aee3-4722-9247-379e9975fb4f None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:25.260222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:25.304906 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becd8ce9-3477-465c-a6c9-66b3ef6d1416 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:25.342620 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:25.355593 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6130c4f-508f-48cb-8eb7-df659ebaeec4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:25.356519 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6130c4f-508f-48cb-8eb7-df659ebaeec4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:25.357002 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6130c4f-508f-48cb-8eb7-df659ebaeec4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:25.357497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6130c4f-508f-48cb-8eb7-df659ebaeec4 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:25.358434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6130c4f-508f-48cb-8eb7-df659ebaeec4 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:25.358988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6130c4f-508f-48cb-8eb7-df659ebaeec4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:25.361397 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6130c4f-508f-48cb-8eb7-df659ebaeec4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:25.368243 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 478/956] 10.222.0.49 () {56 vars in 1122 bytes} [Fri Sep 16 03:28:25 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:25.373368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff32d7b-aee3-4722-9247-379e9975fb4f None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:25.374506 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6130c4f-508f-48cb-8eb7-df659ebaeec4 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:25.411890 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cef349-8a3a-413d-a131-f83c6fad08c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:25.412538 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cef349-8a3a-413d-a131-f83c6fad08c8 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:25.412820 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cef349-8a3a-413d-a131-f83c6fad08c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:25.413093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cef349-8a3a-413d-a131-f83c6fad08c8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:25.420258 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99cef349-8a3a-413d-a131-f83c6fad08c8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:25.422952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99cef349-8a3a-413d-a131-f83c6fad08c8 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:25.436138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-99cef349-8a3a-413d-a131-f83c6fad08c8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:25.436138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:25.436138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:25.436138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:25.436138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:25.436138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:25.436138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:25.436138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:25.436138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:25.436138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:25.436138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:25.436138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:25.436138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:28:25.436138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:28:25.436138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:28:25.437115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:28:25.437115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:25.437115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:25.437115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:25.437115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:25.437115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:25.437115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:25.437115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:25.437115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:25.437115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:25.437115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:25.437115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:25.437115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:25.437115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:25.437115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:25.437115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:25.437115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:25.438325 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:28:25.438325 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:28:25.438325 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:28:25.438325 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:28:25.438325 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:28:25.438325 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:28:25.438325 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:25.438325 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:25.442163 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 479/957] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:28:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:25.443181 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cef349-8a3a-413d-a131-f83c6fad08c8 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:25.494494 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6130c4f-508f-48cb-8eb7-df659ebaeec4 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:25.513566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:25.538208 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca98380-f272-4db0-9d94-3b458451c59c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:25.539282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca98380-f272-4db0-9d94-3b458451c59c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:25.540329 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca98380-f272-4db0-9d94-3b458451c59c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:25.540869 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca98380-f272-4db0-9d94-3b458451c59c None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:25.542065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca98380-f272-4db0-9d94-3b458451c59c None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:25.542596 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca98380-f272-4db0-9d94-3b458451c59c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:25.544296 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca98380-f272-4db0-9d94-3b458451c59c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:25.552271 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 479/958] 10.222.0.49 () {56 vars in 1122 bytes} [Fri Sep 16 03:28:25 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:25.554020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cef349-8a3a-413d-a131-f83c6fad08c8 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:25.559960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca98380-f272-4db0-9d94-3b458451c59c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:25.596996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3369b8e7-f205-47a2-a267-172eb722c3e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:25.597605 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3369b8e7-f205-47a2-a267-172eb722c3e6 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:25.597884 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3369b8e7-f205-47a2-a267-172eb722c3e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:25.598154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3369b8e7-f205-47a2-a267-172eb722c3e6 None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:25.598808 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3369b8e7-f205-47a2-a267-172eb722c3e6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:25.599137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3369b8e7-f205-47a2-a267-172eb722c3e6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:25.600567 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3369b8e7-f205-47a2-a267-172eb722c3e6 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:25.623273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 480/959] 10.222.0.49 () {56 vars in 1112 bytes} [Fri Sep 16 03:28:25 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:25.631629 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:25.691595 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca98380-f272-4db0-9d94-3b458451c59c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:25.732036 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb36530-3dc3-4285-b58e-eedf48986c47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:25.732658 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb36530-3dc3-4285-b58e-eedf48986c47 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:25.732934 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb36530-3dc3-4285-b58e-eedf48986c47 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:25.733204 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb36530-3dc3-4285-b58e-eedf48986c47 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:25.733877 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb36530-3dc3-4285-b58e-eedf48986c47 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:25.735834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb36530-3dc3-4285-b58e-eedf48986c47 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:25.816981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 480/960] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:28:25 2022] POST /identity/v3/endpoints => generated 349 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:28:26.847545 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fcfb4d-0e43-4196-8f06-c114234fae2f None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:26.848112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fcfb4d-0e43-4196-8f06-c114234fae2f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:26.848112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fcfb4d-0e43-4196-8f06-c114234fae2f None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:26.849792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 481/961] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:26.855907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9eb3b4-069d-489e-a7e9-2b1272c72bf2 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:26.856431 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9eb3b4-069d-489e-a7e9-2b1272c72bf2 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:26.856431 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9eb3b4-069d-489e-a7e9-2b1272c72bf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:26.876923 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2b9eb3b4-069d-489e-a7e9-2b1272c72bf2 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:26.878172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2b9eb3b4-069d-489e-a7e9-2b1272c72bf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:26.956656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 481/962] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:26 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:26.974418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5009cef4-0e6d-4a64-8b84-33071b32f47b None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:26.974742 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5009cef4-0e6d-4a64-8b84-33071b32f47b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:26.975016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5009cef4-0e6d-4a64-8b84-33071b32f47b None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:26.976602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 482/963] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:26.981892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed35bde-d9cb-47a2-8c19-ba639c3fd843 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:26.982183 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed35bde-d9cb-47a2-8c19-ba639c3fd843 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:26.982457 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed35bde-d9cb-47a2-8c19-ba639c3fd843 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:27.004157 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-0ed35bde-d9cb-47a2-8c19-ba639c3fd843 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:27.005336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0ed35bde-d9cb-47a2-8c19-ba639c3fd843 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:27.033562 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:27.075555 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 482/964] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:26 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:27.226539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5009cef4-0e6d-4a64-8b84-33071b32f47b None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:27.331585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5009cef4-0e6d-4a64-8b84-33071b32f47b None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:27.360037 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed35bde-d9cb-47a2-8c19-ba639c3fd843 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:27.361759 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:27.371159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f36df08-388a-49d4-ab58-cf1638c2c986 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:27.371776 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f36df08-388a-49d4-ab58-cf1638c2c986 None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:27.377866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f36df08-388a-49d4-ab58-cf1638c2c986 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:27.377866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f36df08-388a-49d4-ab58-cf1638c2c986 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:27.377866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f36df08-388a-49d4-ab58-cf1638c2c986 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:27.377866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f36df08-388a-49d4-ab58-cf1638c2c986 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:27.460562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 483/965] 10.222.0.49 () {60 vars in 1141 bytes} [Fri Sep 16 03:28:27 2022] POST /identity/v3/services => generated 242 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:28:27.479369 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed35bde-d9cb-47a2-8c19-ba639c3fd843 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:27.526951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c2dfc5-83c2-4a61-b0ee-51c6d3120b6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:27.527615 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c2dfc5-83c2-4a61-b0ee-51c6d3120b6f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:27.527898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c2dfc5-83c2-4a61-b0ee-51c6d3120b6f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:27.528172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c2dfc5-83c2-4a61-b0ee-51c6d3120b6f None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:27.541415 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c2dfc5-83c2-4a61-b0ee-51c6d3120b6f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:28:27.546401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-53c2dfc5-83c2-4a61-b0ee-51c6d3120b6f None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 03:28:27.546401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:27.546401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:27.546401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:27.546401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:27.546401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:27.546401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:27.546401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:27.546401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:27.546401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:27.546401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:27.546401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:27.546401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:28:27.546401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:28:27.546401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:28:27.546401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:28:27.548216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:28:27.548216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:28:27.548216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:28:27.548216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:28:27.548216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:27.548216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:27.548216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:27.548216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:27.548216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:27.548216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:27.548216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:27.548216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:27.548216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:27.548216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:27.548216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:27.548216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:27.549071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:27.549071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:27.549071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:27.549071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:27.549071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:28:27.549071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:28:27.549071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:28:27.549071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:28:27.549071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:28:27.549071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:28:27.549071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 03:28:27.549071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:27.549071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 483/966] 10.222.0.49 () {56 vars in 1120 bytes} [Fri Sep 16 03:28:27 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:27.554769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f36df08-388a-49d4-ab58-cf1638c2c986 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:27.681266 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f36df08-388a-49d4-ab58-cf1638c2c986 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:27.721162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7fde794-4369-4853-8461-bf4aba5c1c70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:27.721787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7fde794-4369-4853-8461-bf4aba5c1c70 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:27.722070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7fde794-4369-4853-8461-bf4aba5c1c70 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:27.722341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7fde794-4369-4853-8461-bf4aba5c1c70 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:27.723115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7fde794-4369-4853-8461-bf4aba5c1c70 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:27.723421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7fde794-4369-4853-8461-bf4aba5c1c70 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:27.724795 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7fde794-4369-4853-8461-bf4aba5c1c70 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:27.734091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 484/967] 10.222.0.49 () {56 vars in 1115 bytes} [Fri Sep 16 03:28:27 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:27.739753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c2dfc5-83c2-4a61-b0ee-51c6d3120b6f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:27.764257 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:27.851706 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c2dfc5-83c2-4a61-b0ee-51c6d3120b6f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:27.888137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10271474-612d-49d0-aae7-15318a83fdf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:27.889557 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10271474-612d-49d0-aae7-15318a83fdf1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:27.890048 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10271474-612d-49d0-aae7-15318a83fdf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:27.891283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10271474-612d-49d0-aae7-15318a83fdf1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:27.902635 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10271474-612d-49d0-aae7-15318a83fdf1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:27.904493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10271474-612d-49d0-aae7-15318a83fdf1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:27.920578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-10271474-612d-49d0-aae7-15318a83fdf1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:27.920578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:27.920578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:27.920578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:27.920578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:27.920578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:27.920578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:27.920578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:27.920578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:27.920578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:27.920578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:27.920578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:27.920578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:27.920578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:27.920578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:27.920578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:27.922156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:27.922156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:27.922156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:27.922156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:27.922156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:27.922156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:27.922156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:27.922156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:27.922156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:27.922156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:27.922156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:27.922156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:27.922156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:27.922156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:27.922156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:27.922156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:27.922156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:27.923129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:27.923129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:27.923129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:27.925202 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 484/968] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:27.931100 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7fde794-4369-4853-8461-bf4aba5c1c70 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:27.943691 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6607544a-ef13-4590-9a6a-1f0cab4ff702 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:27.944038 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6607544a-ef13-4590-9a6a-1f0cab4ff702 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:27.944241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6607544a-ef13-4590-9a6a-1f0cab4ff702 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:27.945804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 485/969] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:28:27.950761 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc927cb-160e-40e6-8df1-505007031a89 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:27.951079 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc927cb-160e-40e6-8df1-505007031a89 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:27.951353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc927cb-160e-40e6-8df1-505007031a89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:27.971209 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8cc927cb-160e-40e6-8df1-505007031a89 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:27.972493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8cc927cb-160e-40e6-8df1-505007031a89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:27.995049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:28.055838 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 486/970] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:27 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 03:28:28.064869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7fde794-4369-4853-8461-bf4aba5c1c70 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:28.100350 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a6c618-ef4d-4d39-81e0-1fb9f9ea3565 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:28.101364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a6c618-ef4d-4d39-81e0-1fb9f9ea3565 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:28.101854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a6c618-ef4d-4d39-81e0-1fb9f9ea3565 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:28.102336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a6c618-ef4d-4d39-81e0-1fb9f9ea3565 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:28.103220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a6c618-ef4d-4d39-81e0-1fb9f9ea3565 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:28.103747 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a6c618-ef4d-4d39-81e0-1fb9f9ea3565 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:28.105326 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a6c618-ef4d-4d39-81e0-1fb9f9ea3565 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:28.118935 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 485/971] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:28.123709 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf790f13-43f0-4cd4-85c5-430ea63607fc None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:28.124004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf790f13-43f0-4cd4-85c5-430ea63607fc None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:28.124283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf790f13-43f0-4cd4-85c5-430ea63607fc None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:28.125393 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a6c618-ef4d-4d39-81e0-1fb9f9ea3565 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:28.126374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 487/972] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:28.131238 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47412ada-3163-4ed0-971a-7b577144179c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:28.131520 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47412ada-3163-4ed0-971a-7b577144179c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:28.131793 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47412ada-3163-4ed0-971a-7b577144179c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:28.138194 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:28.152664 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-47412ada-3163-4ed0-971a-7b577144179c None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:28.154541 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-47412ada-3163-4ed0-971a-7b577144179c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:28.233634 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 488/973] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:28 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:28.253344 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a6c618-ef4d-4d39-81e0-1fb9f9ea3565 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:28.263205 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:28.289520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac3177f-328d-459f-813f-65a9b487e6e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:28.290163 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac3177f-328d-459f-813f-65a9b487e6e9 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:28.290448 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac3177f-328d-459f-813f-65a9b487e6e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:28.290726 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac3177f-328d-459f-813f-65a9b487e6e9 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:28.298804 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac3177f-328d-459f-813f-65a9b487e6e9 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:28.300447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac3177f-328d-459f-813f-65a9b487e6e9 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:28.313364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-cac3177f-328d-459f-813f-65a9b487e6e9 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 03:28:28.313364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:28.313364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:28.313364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:28.313364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:28.313364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:28.313364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:28.313364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:28.313364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:28.313364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:28.313364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:28.313364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:28.313364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:28:28.313364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:28:28.313364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:28:28.313364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:28:28.314514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:28.314514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:28.314514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:28:28.314514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:28.314514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:28:28.314514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:28.314514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:28.314514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:28.314514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:28.314514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:28.314514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:28.314514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:28.314514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:28.314514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:28.314514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:28.314514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:28.314514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:28.314514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:28.316111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:28.316111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:28.316111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:28:28.316111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:28:28.316111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:28:28.316111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:28:28.316111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:28:28.316111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:28:28.316111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 03:28:28.316111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:28.316111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 486/974] 10.222.0.49 () {56 vars in 1117 bytes} [Fri Sep 16 03:28:28 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:28.320585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47412ada-3163-4ed0-971a-7b577144179c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:28.340152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac3177f-328d-459f-813f-65a9b487e6e9 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:28.448980 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac3177f-328d-459f-813f-65a9b487e6e9 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:28.463016 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47412ada-3163-4ed0-971a-7b577144179c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:28.484632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bda02fb-8497-4b35-aa1d-f23c0225a188 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:28.485725 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bda02fb-8497-4b35-aa1d-f23c0225a188 None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:28.486205 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bda02fb-8497-4b35-aa1d-f23c0225a188 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:28.486671 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bda02fb-8497-4b35-aa1d-f23c0225a188 None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:28.487551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bda02fb-8497-4b35-aa1d-f23c0225a188 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:28.489180 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bda02fb-8497-4b35-aa1d-f23c0225a188 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:28.502011 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:28.513151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8058fe1-1e6b-44c4-8a6e-351d9462ae66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:28.513151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8058fe1-1e6b-44c4-8a6e-351d9462ae66 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:28.513151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8058fe1-1e6b-44c4-8a6e-351d9462ae66 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:28.513151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8058fe1-1e6b-44c4-8a6e-351d9462ae66 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:28.513505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8058fe1-1e6b-44c4-8a6e-351d9462ae66 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:28.513505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8058fe1-1e6b-44c4-8a6e-351d9462ae66 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:28.513505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8058fe1-1e6b-44c4-8a6e-351d9462ae66 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:28.534670 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 489/975] 10.222.0.49 () {56 vars in 1149 bytes} [Fri Sep 16 03:28:28 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:28.534670 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8058fe1-1e6b-44c4-8a6e-351d9462ae66 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:28.536223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.sql.core [None req-5bda02fb-8497-4b35-aa1d-f23c0225a188 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 16 03:28:28.536223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 16 03:28:28.536223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [parameters: {'user_id': 'acdb4d2c4aae4cb1b29bae94868a7c7d', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 16 03:28:28.536223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81761) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 16 03:28:28.543439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-5bda02fb-8497-4b35-aa1d-f23c0225a188 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 16 03:28:28.543439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:28.543439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 16 03:28:28.543439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 16 03:28:28.543439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 16 03:28:28.543439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 16 03:28:28.543439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 16 03:28:28.543439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application result = self._query(query) Sep 16 03:28:28.543439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 16 03:28:28.543439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application conn.query(q) Sep 16 03:28:28.543439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 16 03:28:28.543439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 16 03:28:28.543439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 16 03:28:28.543439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application result.read() Sep 16 03:28:28.543439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 16 03:28:28.548486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 16 03:28:28.548486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 16 03:28:28.548486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 16 03:28:28.548486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 16 03:28:28.548486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 16 03:28:28.548486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 16 03:28:28.548486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 16 03:28:28.548486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 16 03:28:28.548486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:28.548486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 16 03:28:28.548486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:28.548486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:28.548486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 16 03:28:28.548486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 16 03:28:28.548486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 16 03:28:28.548486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 16 03:28:28.548486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 16 03:28:28.548486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application next(self.gen) Sep 16 03:28:28.549468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 16 03:28:28.549468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application yield resource Sep 16 03:28:28.549468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 16 03:28:28.549468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application next(self.gen) Sep 16 03:28:28.549468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 16 03:28:28.549468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self.session.rollback() Sep 16 03:28:28.549468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 16 03:28:28.549468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self.force_reraise() Sep 16 03:28:28.549468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 16 03:28:28.549468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise self.value Sep 16 03:28:28.549468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 16 03:28:28.549468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 16 03:28:28.549468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 16 03:28:28.549468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application session.commit() Sep 16 03:28:28.549468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 16 03:28:28.549468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 16 03:28:28.549468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 16 03:28:28.550376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._prepare_impl() Sep 16 03:28:28.550376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 16 03:28:28.550376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self.session.flush() Sep 16 03:28:28.550376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 16 03:28:28.550376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._flush(objects) Sep 16 03:28:28.550376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 16 03:28:28.550376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 16 03:28:28.550376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 16 03:28:28.550376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application compat.raise_( Sep 16 03:28:28.550376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 16 03:28:28.550376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception Sep 16 03:28:28.550376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 16 03:28:28.550376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application flush_context.execute() Sep 16 03:28:28.550376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 16 03:28:28.550376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rec.execute(self) Sep 16 03:28:28.550376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 16 03:28:28.550376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 16 03:28:28.555553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 16 03:28:28.555553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 16 03:28:28.555553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 16 03:28:28.555553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 16 03:28:28.555553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 16 03:28:28.555553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 16 03:28:28.555553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 16 03:28:28.555553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 16 03:28:28.555553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 16 03:28:28.555553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 16 03:28:28.555553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 16 03:28:28.555553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 16 03:28:28.555553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 16 03:28:28.555553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 16 03:28:28.555553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 16 03:28:28.555553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception Sep 16 03:28:28.557563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 16 03:28:28.557563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 16 03:28:28.557563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 16 03:28:28.557563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 16 03:28:28.557563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 16 03:28:28.557563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application result = self._query(query) Sep 16 03:28:28.557563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 16 03:28:28.557563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application conn.query(q) Sep 16 03:28:28.557563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 16 03:28:28.557563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 16 03:28:28.557563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 16 03:28:28.557563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application result.read() Sep 16 03:28:28.557563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 16 03:28:28.557563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 16 03:28:28.557563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 16 03:28:28.557563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 16 03:28:28.559792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 16 03:28:28.559792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 16 03:28:28.559792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 16 03:28:28.559792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 16 03:28:28.559792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 16 03:28:28.559792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 16 03:28:28.559792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [parameters: {'user_id': 'acdb4d2c4aae4cb1b29bae94868a7c7d', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 16 03:28:28.559792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 16 03:28:28.559792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:28.559792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:28.559792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:28.559792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:28.559792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:28.559792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:28.559792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:28.561912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:28.561912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:28.561912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:28.561912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:28.561912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:28.561912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:28.561912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:28.561912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 16 03:28:28.561912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 16 03:28:28.561912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:28.561912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:28.561912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:28:28.561912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:28.561912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:28:28.561912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:28.561912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 16 03:28:28.561912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 16 03:28:28.565001 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 16 03:28:28.565001 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 16 03:28:28.565001 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 16 03:28:28.565001 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 16 03:28:28.565001 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 16 03:28:28.565001 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:28.565001 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 487/976] 10.222.0.49 () {60 vars in 1131 bytes} [Fri Sep 16 03:28:28 2022] POST /identity/v3/users => generated 160 bytes in 205 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:28.565001 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bda02fb-8497-4b35-aa1d-f23c0225a188 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:28.579694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:28.638995 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8058fe1-1e6b-44c4-8a6e-351d9462ae66 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:28.654870 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bda02fb-8497-4b35-aa1d-f23c0225a188 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:28.676556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7084ab8d-5609-4421-a892-7c439ef0591d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:28.677283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7084ab8d-5609-4421-a892-7c439ef0591d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:28.677637 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7084ab8d-5609-4421-a892-7c439ef0591d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:28.677983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7084ab8d-5609-4421-a892-7c439ef0591d None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:28.678639 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7084ab8d-5609-4421-a892-7c439ef0591d None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:28.680091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7084ab8d-5609-4421-a892-7c439ef0591d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:28.689549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-7084ab8d-5609-4421-a892-7c439ef0591d None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 03:28:28.689549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:28.689549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:28.689549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:28.689549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:28.689549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:28.689549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:28.689549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:28.689549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 03:28:28.689549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:28.689549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:28.689549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:28.689549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:28.689549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:28.689549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:28.689549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:28.689549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:28.690848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:28.690848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:28.690848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:28.690848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:28.690848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:28.690848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:28.690848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:28.690848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:28.690848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:28.690848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:28.690848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:28.690848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:28.690848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:28.690848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:28.690848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:28.690848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:28.691945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:28.691945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:28.691945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:28.691945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:28.691945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:28.691945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:28.691945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:28.691945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:28.691945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:28.691945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:28.691945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:28.691945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:28.691945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 03:28:28.691945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:28.692981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 490/977] 10.222.0.49 () {56 vars in 1126 bytes} [Fri Sep 16 03:28:28 2022] GET /identity/v3/services/object-store => generated 93 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:28.693391 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7084ab8d-5609-4421-a892-7c439ef0591d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:28.698641 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19d5e97-bbd7-4d62-bb8b-825cfae91aaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:28.699293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19d5e97-bbd7-4d62-bb8b-825cfae91aaf None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:28.699569 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19d5e97-bbd7-4d62-bb8b-825cfae91aaf None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:28.699834 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19d5e97-bbd7-4d62-bb8b-825cfae91aaf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:28.751687 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:28.773421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19d5e97-bbd7-4d62-bb8b-825cfae91aaf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:28.775037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19d5e97-bbd7-4d62-bb8b-825cfae91aaf None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:28.802500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7084ab8d-5609-4421-a892-7c439ef0591d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:28.806236 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:28.840248 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bb0cce-f9de-4806-80d4-99a56494dba9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:28.840967 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bb0cce-f9de-4806-80d4-99a56494dba9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:28.841313 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bb0cce-f9de-4806-80d4-99a56494dba9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:28.841662 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bb0cce-f9de-4806-80d4-99a56494dba9 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:28.849935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bb0cce-f9de-4806-80d4-99a56494dba9 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:28.851414 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bb0cce-f9de-4806-80d4-99a56494dba9 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:28.862805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-53bb0cce-f9de-4806-80d4-99a56494dba9 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 16 03:28:28.862805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:28.862805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:28.862805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:28.862805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:28.862805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:28.862805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:28.862805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:28.862805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:28.862805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:28.862805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:28.862805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:28.862805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:28:28.862805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:28:28.862805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:28:28.862805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:28:28.863816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:28.863816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:28.863816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:28:28.863816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:28.863816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:28:28.863816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:28.863816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:28.863816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:28.863816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:28.863816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:28.863816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:28.863816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:28.863816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:28.863816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:28.863816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:28.863816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:28.863816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:28.863816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:28.864789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:28.864789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:28.864789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:28:28.864789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:28:28.864789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:28:28.864789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:28:28.864789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:28:28.864789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:28:28.864789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 16 03:28:28.864789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:28.864789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 491/978] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:28:28 2022] GET /identity/v3/users/cinder => generated 84 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:28.864789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bb0cce-f9de-4806-80d4-99a56494dba9 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:28.885594 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 488/979] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:28:28 2022] GET /identity/v3/auth/tokens => generated 1570 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 03:28:28.886415 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19d5e97-bbd7-4d62-bb8b-825cfae91aaf None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:28.977738 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bb0cce-f9de-4806-80d4-99a56494dba9 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:28.992270 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:28.994105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19d5e97-bbd7-4d62-bb8b-825cfae91aaf None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:29.014170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeea510-309e-44bf-927c-5e091734d37e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:29.014775 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeea510-309e-44bf-927c-5e091734d37e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:29.015124 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeea510-309e-44bf-927c-5e091734d37e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:29.015598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeea510-309e-44bf-927c-5e091734d37e None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:29.016515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eeea510-309e-44bf-927c-5e091734d37e None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:29.016824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eeea510-309e-44bf-927c-5e091734d37e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:29.018243 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eeea510-309e-44bf-927c-5e091734d37e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:29.024624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 492/980] 10.222.0.49 () {56 vars in 1122 bytes} [Fri Sep 16 03:28:28 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:29.025336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeea510-309e-44bf-927c-5e091734d37e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:29.025675 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:29.033075 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58534940-7ba6-403d-a34c-1bfa368763e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:29.034535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58534940-7ba6-403d-a34c-1bfa368763e7 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:29.034861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58534940-7ba6-403d-a34c-1bfa368763e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:29.035147 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58534940-7ba6-403d-a34c-1bfa368763e7 None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:29.035841 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58534940-7ba6-403d-a34c-1bfa368763e7 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:29.040741 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58534940-7ba6-403d-a34c-1bfa368763e7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:29.040741 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58534940-7ba6-403d-a34c-1bfa368763e7 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:29.086937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 489/981] 10.222.0.49 () {56 vars in 1101 bytes} [Fri Sep 16 03:28:28 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:29.086937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58534940-7ba6-403d-a34c-1bfa368763e7 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:29.147376 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeea510-309e-44bf-927c-5e091734d37e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:29.187200 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b9cd68-df47-4e8a-ad37-60ee50bb9337 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:29.188309 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b9cd68-df47-4e8a-ad37-60ee50bb9337 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:29.188593 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b9cd68-df47-4e8a-ad37-60ee50bb9337 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:29.188867 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b9cd68-df47-4e8a-ad37-60ee50bb9337 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:29.201299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b9cd68-df47-4e8a-ad37-60ee50bb9337 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:29.202788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b9cd68-df47-4e8a-ad37-60ee50bb9337 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:29.214855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58534940-7ba6-403d-a34c-1bfa368763e7 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:29.220676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-f8b9cd68-df47-4e8a-ad37-60ee50bb9337 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:29.220676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:29.220676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:29.220676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:29.220676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:29.220676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:29.220676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:29.220676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:29.220676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:29.220676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:29.220676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:29.220676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:29.220676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:29.220676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:29.220676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:29.220676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:29.221668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:29.221668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:29.221668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:29.221668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:29.221668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:29.221668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:29.221668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:29.221668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:29.221668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:29.221668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:29.221668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:29.221668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:29.221668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:29.221668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:29.221668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:29.221668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:29.221668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:29.222564 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:29.222564 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:29.222564 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:29.226659 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 493/982] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:29.227719 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b9cd68-df47-4e8a-ad37-60ee50bb9337 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:29.258023 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3cc0c4-7f2e-4b5f-b15e-291213478e5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:29.259137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3cc0c4-7f2e-4b5f-b15e-291213478e5d None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:29.259725 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3cc0c4-7f2e-4b5f-b15e-291213478e5d None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:29.260005 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3cc0c4-7f2e-4b5f-b15e-291213478e5d None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:29.260645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3cc0c4-7f2e-4b5f-b15e-291213478e5d None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:29.261199 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3cc0c4-7f2e-4b5f-b15e-291213478e5d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:29.263381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3cc0c4-7f2e-4b5f-b15e-291213478e5d None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:29.264549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:29.272623 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 490/983] 10.222.0.49 () {56 vars in 1122 bytes} [Fri Sep 16 03:28:29 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:29.281500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3cc0c4-7f2e-4b5f-b15e-291213478e5d None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:29.350176 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:29.360099 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b9cd68-df47-4e8a-ad37-60ee50bb9337 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:29.413034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8471aa-848d-4fef-a5ab-166b1b15845f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:29.413738 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8471aa-848d-4fef-a5ab-166b1b15845f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:29.414085 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8471aa-848d-4fef-a5ab-166b1b15845f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:29.414963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8471aa-848d-4fef-a5ab-166b1b15845f None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:29.415105 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8471aa-848d-4fef-a5ab-166b1b15845f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:29.415662 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8471aa-848d-4fef-a5ab-166b1b15845f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:29.416787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3cc0c4-7f2e-4b5f-b15e-291213478e5d None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:29.417018 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8471aa-848d-4fef-a5ab-166b1b15845f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:29.434584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 494/984] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:29.440724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8471aa-848d-4fef-a5ab-166b1b15845f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:29.462165 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ce5192-1ed0-4f32-9b11-a7082a761e11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:29.462957 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ce5192-1ed0-4f32-9b11-a7082a761e11 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:29.463246 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ce5192-1ed0-4f32-9b11-a7082a761e11 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:29.463522 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ce5192-1ed0-4f32-9b11-a7082a761e11 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:29.464540 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ce5192-1ed0-4f32-9b11-a7082a761e11 None admin] RBAC: Adding query filter params (interface=admin, service_id=d0aefe94b2db42b7a4c4ea33267a862f, region_id=RegionOne) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:29.464853 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ce5192-1ed0-4f32-9b11-a7082a761e11 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:29.466222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ce5192-1ed0-4f32-9b11-a7082a761e11 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:29.473533 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 491/985] 10.222.0.49 () {56 vars in 1249 bytes} [Fri Sep 16 03:28:29 2022] GET /identity/v3/endpoints?service_id=d0aefe94b2db42b7a4c4ea33267a862f&interface=admin®ion_id=RegionOne => generated 193 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:29.479295 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ce5192-1ed0-4f32-9b11-a7082a761e11 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:29.541914 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:29.553377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8471aa-848d-4fef-a5ab-166b1b15845f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:29.586790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ce5192-1ed0-4f32-9b11-a7082a761e11 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:29.591716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb67adf-503f-42fc-b1a4-758d81c80e7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:29.592344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb67adf-503f-42fc-b1a4-758d81c80e7d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:29.592621 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb67adf-503f-42fc-b1a4-758d81c80e7d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:29.592890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb67adf-503f-42fc-b1a4-758d81c80e7d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:29.599775 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb67adf-503f-42fc-b1a4-758d81c80e7d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:29.601205 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb67adf-503f-42fc-b1a4-758d81c80e7d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:29.603903 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:29.613512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-abb67adf-503f-42fc-b1a4-758d81c80e7d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:29.613512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:29.613512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:29.613512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:29.613512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:29.613512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:29.613512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:29.613512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:29.613512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:29.613512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:29.613512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:29.613512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:29.613512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:28:29.613512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:28:29.613512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:28:29.618353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:28:29.618353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:29.618353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:29.618353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:29.618353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:29.618353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:29.618353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:29.618353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:29.618353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:29.618353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:29.618353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:29.618353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:29.618353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:29.618353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:29.618353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:29.618353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:29.618353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:29.619301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:28:29.619301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:28:29.619301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:28:29.619301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:28:29.619301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:28:29.619301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:28:29.619301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:29.619301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:29.619301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 495/986] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:28:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:29.621234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb67adf-503f-42fc-b1a4-758d81c80e7d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:29.627122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5638ca8a-4ce1-4690-a99c-6ae6d538d359 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:29.627760 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5638ca8a-4ce1-4690-a99c-6ae6d538d359 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:29.628047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5638ca8a-4ce1-4690-a99c-6ae6d538d359 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:29.628322 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5638ca8a-4ce1-4690-a99c-6ae6d538d359 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:29.628943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5638ca8a-4ce1-4690-a99c-6ae6d538d359 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:29.630371 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5638ca8a-4ce1-4690-a99c-6ae6d538d359 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:29.637499 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 492/987] 10.222.0.49 () {56 vars in 1087 bytes} [Fri Sep 16 03:28:29 2022] GET /identity/v3/services => generated 1727 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:28:29.735236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb67adf-503f-42fc-b1a4-758d81c80e7d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:29.739340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e14d7f-7f44-40be-a512-41d909e1c6f8 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:29.739637 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e14d7f-7f44-40be-a512-41d909e1c6f8 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:29.739904 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e14d7f-7f44-40be-a512-41d909e1c6f8 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:29.741545 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 493/988] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:28:29.747640 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979ff47a-dc68-40c3-8062-675a570c9a30 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:29.748115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979ff47a-dc68-40c3-8062-675a570c9a30 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:29.748403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979ff47a-dc68-40c3-8062-675a570c9a30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:29.768531 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-979ff47a-dc68-40c3-8062-675a570c9a30 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:29.770129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-979ff47a-dc68-40c3-8062-675a570c9a30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:29.771584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064b217e-b8a2-459e-9181-a27b14e8fbb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:29.772220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064b217e-b8a2-459e-9181-a27b14e8fbb1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:29.772499 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064b217e-b8a2-459e-9181-a27b14e8fbb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:29.772771 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064b217e-b8a2-459e-9181-a27b14e8fbb1 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:29.773471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064b217e-b8a2-459e-9181-a27b14e8fbb1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:29.773777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064b217e-b8a2-459e-9181-a27b14e8fbb1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:29.775254 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064b217e-b8a2-459e-9181-a27b14e8fbb1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:29.789236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 496/989] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:28:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:29.795297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064b217e-b8a2-459e-9181-a27b14e8fbb1 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:29.809280 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:29.851098 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 494/990] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:29 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:29.907205 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064b217e-b8a2-459e-9181-a27b14e8fbb1 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:29.941494 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a20e0b-3891-47fa-a82e-c4028f913fcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:29.942150 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a20e0b-3891-47fa-a82e-c4028f913fcd None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:29.942426 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a20e0b-3891-47fa-a82e-c4028f913fcd None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:29.942698 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a20e0b-3891-47fa-a82e-c4028f913fcd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:29.943500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a20e0b-3891-47fa-a82e-c4028f913fcd None admin] RBAC: Adding query filter params (role.id=4b248d39ed014678bfc26b6f0acd7134, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=d40810f61ead4c3da414194b325da80a) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:29.943808 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a20e0b-3891-47fa-a82e-c4028f913fcd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:29.945506 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a20e0b-3891-47fa-a82e-c4028f913fcd None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:29.958996 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 497/991] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:28:29 2022] GET /identity/v3/role_assignments?user.id=d40810f61ead4c3da414194b325da80a&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=4b248d39ed014678bfc26b6f0acd7134 => generated 608 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:30.121157 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979ff47a-dc68-40c3-8062-675a570c9a30 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:30.224310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979ff47a-dc68-40c3-8062-675a570c9a30 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:30.244145 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:30.275290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c87b29-bb09-4825-aa2a-2cb3675dabbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:30.276783 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c87b29-bb09-4825-aa2a-2cb3675dabbf None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:30.277440 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c87b29-bb09-4825-aa2a-2cb3675dabbf None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:30.278179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c87b29-bb09-4825-aa2a-2cb3675dabbf None admin] PATH_INFO: `/v3/services/compute` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:30.279662 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c87b29-bb09-4825-aa2a-2cb3675dabbf None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:30.282091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c87b29-bb09-4825-aa2a-2cb3675dabbf None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:30.294423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-43c87b29-bb09-4825-aa2a-2cb3675dabbf None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 16 03:28:30.294423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:30.294423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:30.294423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:30.294423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:30.294423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:30.294423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:30.294423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:30.294423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 16 03:28:30.294423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:30.294423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:30.294423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:30.294423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:30.294423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:30.294423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:30.294423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:30.294423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:30.295456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:30.295456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:30.295456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:30.295456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:30.295456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:30.295456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:30.295456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:30.295456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:30.295456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:30.295456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:30.295456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:30.295456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:30.295456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:30.295456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:30.295456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:30.295456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:30.296300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:30.296300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:30.296300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:30.296300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:30.296300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:30.296300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:30.296300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:30.296300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:30.296300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:30.296300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:30.296300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:30.296300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:30.296300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 16 03:28:30.296300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:30.298282 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 495/992] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:28:30 2022] GET /identity/v3/services/compute => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:30.302979 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a20e0b-3891-47fa-a82e-c4028f913fcd None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:30.389729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:30.414019 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a20e0b-3891-47fa-a82e-c4028f913fcd None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:30.448813 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dace79-d3db-4f60-bf46-a8a7c0123f1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:30.449415 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dace79-d3db-4f60-bf46-a8a7c0123f1e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:30.449692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dace79-d3db-4f60-bf46-a8a7c0123f1e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:30.449964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dace79-d3db-4f60-bf46-a8a7c0123f1e None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:30.450588 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dace79-d3db-4f60-bf46-a8a7c0123f1e None admin] RBAC: Adding query filter params (name=compute) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:30.450888 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dace79-d3db-4f60-bf46-a8a7c0123f1e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:30.452261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dace79-d3db-4f60-bf46-a8a7c0123f1e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:30.461998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 498/993] 10.222.0.49 () {56 vars in 1112 bytes} [Fri Sep 16 03:28:30 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:30.462957 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c87b29-bb09-4825-aa2a-2cb3675dabbf None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:30.568276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c87b29-bb09-4825-aa2a-2cb3675dabbf None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:30.609638 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f606d061-c13b-40e1-9bbd-a7d91eee8a26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:30.610529 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f606d061-c13b-40e1-9bbd-a7d91eee8a26 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:30.610835 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f606d061-c13b-40e1-9bbd-a7d91eee8a26 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:30.611139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f606d061-c13b-40e1-9bbd-a7d91eee8a26 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:30.611815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f606d061-c13b-40e1-9bbd-a7d91eee8a26 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:30.612076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f606d061-c13b-40e1-9bbd-a7d91eee8a26 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:30.613446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f606d061-c13b-40e1-9bbd-a7d91eee8a26 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:30.623045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 496/994] 10.222.0.49 () {56 vars in 1112 bytes} [Fri Sep 16 03:28:30 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:30.625561 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dace79-d3db-4f60-bf46-a8a7c0123f1e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:30.727618 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dace79-d3db-4f60-bf46-a8a7c0123f1e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:30.762010 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a148b4-ee16-4ef2-80a7-8a36a29e9ec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:30.762621 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a148b4-ee16-4ef2-80a7-8a36a29e9ec3 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:30.762902 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a148b4-ee16-4ef2-80a7-8a36a29e9ec3 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:30.763200 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a148b4-ee16-4ef2-80a7-8a36a29e9ec3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:30.763896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a148b4-ee16-4ef2-80a7-8a36a29e9ec3 None admin] RBAC: Adding query filter params (interface=public, service_id=a63fee8e7acb4234b3002f840056de93, region_id=RegionOne) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:30.764226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a148b4-ee16-4ef2-80a7-8a36a29e9ec3 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:30.765686 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a148b4-ee16-4ef2-80a7-8a36a29e9ec3 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:30.774978 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 499/995] 10.222.0.49 () {56 vars in 1251 bytes} [Fri Sep 16 03:28:30 2022] GET /identity/v3/endpoints?service_id=a63fee8e7acb4234b3002f840056de93&interface=public®ion_id=RegionOne => generated 194 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:30.776700 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f606d061-c13b-40e1-9bbd-a7d91eee8a26 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:30.820109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:30.879973 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f606d061-c13b-40e1-9bbd-a7d91eee8a26 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:30.928308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c674757-d6fb-4713-a74d-8a857b937331 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:30.928925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c674757-d6fb-4713-a74d-8a857b937331 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:30.929207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c674757-d6fb-4713-a74d-8a857b937331 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:30.929482 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c674757-d6fb-4713-a74d-8a857b937331 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:30.930090 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c674757-d6fb-4713-a74d-8a857b937331 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:30.931509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c674757-d6fb-4713-a74d-8a857b937331 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:30.943156 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 497/996] 10.222.0.49 () {56 vars in 1087 bytes} [Fri Sep 16 03:28:30 2022] GET /identity/v3/services => generated 1727 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:28:31.821444 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18e41ef-1749-4560-a838-cf073aa1bbd0 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:31.821989 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18e41ef-1749-4560-a838-cf073aa1bbd0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:31.821989 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18e41ef-1749-4560-a838-cf073aa1bbd0 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:31.823592 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 500/997] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:31.829864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8f7234-9b23-4860-801d-a8cbdc2567c5 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:31.830328 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8f7234-9b23-4860-801d-a8cbdc2567c5 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:31.830403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8f7234-9b23-4860-801d-a8cbdc2567c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:31.848418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-ab8f7234-9b23-4860-801d-a8cbdc2567c5 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:31.849582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ab8f7234-9b23-4860-801d-a8cbdc2567c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:31.914821 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 498/998] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:31 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:32.181761 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18e41ef-1749-4560-a838-cf073aa1bbd0 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:32.184628 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:32.196602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14edb435-1515-41b0-988e-131d766b2349 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:32.196896 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14edb435-1515-41b0-988e-131d766b2349 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:32.197155 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14edb435-1515-41b0-988e-131d766b2349 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:32.198620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 499/999] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:32.203607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462ea77f-1397-4c4b-a0a4-5b29a27ad5c5 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:32.203892 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462ea77f-1397-4c4b-a0a4-5b29a27ad5c5 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:32.204170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462ea77f-1397-4c4b-a0a4-5b29a27ad5c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:32.223782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-462ea77f-1397-4c4b-a0a4-5b29a27ad5c5 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:32.224948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-462ea77f-1397-4c4b-a0a4-5b29a27ad5c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:32.254870 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:32.293894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18e41ef-1749-4560-a838-cf073aa1bbd0 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:32.306354 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 500/1000] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:32 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 03:28:32.335907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3a0ae2-15ef-49b6-a39b-3e0d139e7dd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:32.336538 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3a0ae2-15ef-49b6-a39b-3e0d139e7dd7 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:32.336816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3a0ae2-15ef-49b6-a39b-3e0d139e7dd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:32.337088 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3a0ae2-15ef-49b6-a39b-3e0d139e7dd7 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:32.337661 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3a0ae2-15ef-49b6-a39b-3e0d139e7dd7 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:32.339030 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3a0ae2-15ef-49b6-a39b-3e0d139e7dd7 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:32.348408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-1d3a0ae2-15ef-49b6-a39b-3e0d139e7dd7 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 03:28:32.348408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:32.348408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:32.348408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:32.348408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:32.348408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:32.348408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:32.348408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:32.348408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 03:28:32.348408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:32.348408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:32.348408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:32.348408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:32.348408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:32.348408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:32.348408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:32.348408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:32.349378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:32.349378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:32.349378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:32.349378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:32.349378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:32.349378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:32.349378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:32.349378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:32.349378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:32.349378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:32.349378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:32.349378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:32.349378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:32.349378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:32.349378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:32.349378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:32.353650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:32.353650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:32.353650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:32.353650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:32.353650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:32.353650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:32.353650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:32.353650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:32.353650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:32.353650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:32.353650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:32.353650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:32.353650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 03:28:32.353650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:32.353650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 501/1001] 10.222.0.49 () {56 vars in 1126 bytes} [Fri Sep 16 03:28:32 2022] GET /identity/v3/services/object-store => generated 93 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:32.355753 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462ea77f-1397-4c4b-a0a4-5b29a27ad5c5 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:32.456063 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462ea77f-1397-4c4b-a0a4-5b29a27ad5c5 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:32.489122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400dfb0c-9476-40f9-b075-2fb36ff28855 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:32.489728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400dfb0c-9476-40f9-b075-2fb36ff28855 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:32.490004 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400dfb0c-9476-40f9-b075-2fb36ff28855 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:32.490273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400dfb0c-9476-40f9-b075-2fb36ff28855 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:32.490893 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400dfb0c-9476-40f9-b075-2fb36ff28855 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:32.491219 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400dfb0c-9476-40f9-b075-2fb36ff28855 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:32.492575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400dfb0c-9476-40f9-b075-2fb36ff28855 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:32.502031 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 501/1002] 10.222.0.49 () {56 vars in 1122 bytes} [Fri Sep 16 03:28:32 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:32.503197 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3a0ae2-15ef-49b6-a39b-3e0d139e7dd7 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:32.570289 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:32.577450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400dfb0c-9476-40f9-b075-2fb36ff28855 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:32.606380 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3a0ae2-15ef-49b6-a39b-3e0d139e7dd7 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:32.606850 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:32.647277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992d0835-4393-409b-88ef-aacde62cfd6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:32.647886 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992d0835-4393-409b-88ef-aacde62cfd6d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:32.648164 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992d0835-4393-409b-88ef-aacde62cfd6d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:32.648435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992d0835-4393-409b-88ef-aacde62cfd6d None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:32.649219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992d0835-4393-409b-88ef-aacde62cfd6d None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:32.649518 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992d0835-4393-409b-88ef-aacde62cfd6d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:32.651512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992d0835-4393-409b-88ef-aacde62cfd6d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:32.657390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 502/1003] 10.222.0.49 () {56 vars in 1122 bytes} [Fri Sep 16 03:28:32 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:32.662943 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992d0835-4393-409b-88ef-aacde62cfd6d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:32.689383 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400dfb0c-9476-40f9-b075-2fb36ff28855 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:32.726392 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6104ba7-204f-4725-94aa-4076b322a866 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:32.726969 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6104ba7-204f-4725-94aa-4076b322a866 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:32.727253 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6104ba7-204f-4725-94aa-4076b322a866 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:32.727528 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6104ba7-204f-4725-94aa-4076b322a866 None admin] PATH_INFO: `/v3/services/image` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:32.728101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6104ba7-204f-4725-94aa-4076b322a866 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:32.729461 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6104ba7-204f-4725-94aa-4076b322a866 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:32.738426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-c6104ba7-204f-4725-94aa-4076b322a866 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 16 03:28:32.738426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:32.738426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:32.738426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:32.738426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:32.738426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:32.738426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:32.738426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:32.738426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 16 03:28:32.738426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:32.738426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:32.738426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:32.738426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:32.738426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:32.738426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:32.738426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:32.738426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:32.739645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:32.739645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:32.739645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:32.739645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:32.739645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:32.739645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:32.739645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:32.739645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:32.739645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:32.739645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:32.739645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:32.739645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:32.739645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:32.739645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:32.739645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:32.739645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:32.740701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:32.740701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:32.740701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:32.740701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:32.740701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:32.740701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:32.740701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:32.740701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:32.740701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:32.740701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:32.740701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:32.740701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:32.740701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 16 03:28:32.740701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:32.740701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 502/1004] 10.222.0.49 () {56 vars in 1105 bytes} [Fri Sep 16 03:28:32 2022] GET /identity/v3/services/image => generated 86 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:32.745410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6104ba7-204f-4725-94aa-4076b322a866 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:32.772018 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992d0835-4393-409b-88ef-aacde62cfd6d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:32.803956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:32.810475 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61b16ea-1976-4623-98fa-bb924b787428 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:32.811110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61b16ea-1976-4623-98fa-bb924b787428 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:32.811389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61b16ea-1976-4623-98fa-bb924b787428 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:32.811658 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61b16ea-1976-4623-98fa-bb924b787428 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:32.813151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61b16ea-1976-4623-98fa-bb924b787428 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:32.814649 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61b16ea-1976-4623-98fa-bb924b787428 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:32.837413 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:32.847324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6104ba7-204f-4725-94aa-4076b322a866 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:32.880031 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3875a8d8-06cb-41b7-8256-f49366cdfb7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:32.880637 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3875a8d8-06cb-41b7-8256-f49366cdfb7e None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:32.880915 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3875a8d8-06cb-41b7-8256-f49366cdfb7e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:32.881186 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3875a8d8-06cb-41b7-8256-f49366cdfb7e None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:32.881799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3875a8d8-06cb-41b7-8256-f49366cdfb7e None admin] RBAC: Adding query filter params (name=image) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:32.882096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3875a8d8-06cb-41b7-8256-f49366cdfb7e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:32.883489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3875a8d8-06cb-41b7-8256-f49366cdfb7e None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:32.887997 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 503/1005] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:32 2022] GET /identity/v3/services?name=image => generated 122 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:32.893961 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3875a8d8-06cb-41b7-8256-f49366cdfb7e None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:32.898096 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 503/1006] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:28:32 2022] POST /identity/v3/endpoints => generated 325 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:28:33.058144 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3875a8d8-06cb-41b7-8256-f49366cdfb7e None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:33.102770 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18614fc4-7514-47db-bae7-8a9cd7c62a02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:33.103661 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18614fc4-7514-47db-bae7-8a9cd7c62a02 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:33.103940 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18614fc4-7514-47db-bae7-8a9cd7c62a02 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:33.104208 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18614fc4-7514-47db-bae7-8a9cd7c62a02 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:33.104831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18614fc4-7514-47db-bae7-8a9cd7c62a02 None admin] RBAC: Adding query filter params (type=image) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:33.105129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18614fc4-7514-47db-bae7-8a9cd7c62a02 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:33.106474 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18614fc4-7514-47db-bae7-8a9cd7c62a02 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:33.111231 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 504/1007] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:32 2022] GET /identity/v3/services?type=image => generated 122 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:33.133033 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118e5119-e47b-4235-862b-f8b3a8412252 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:33.133340 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118e5119-e47b-4235-862b-f8b3a8412252 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:33.133657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118e5119-e47b-4235-862b-f8b3a8412252 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:33.135067 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 504/1008] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:33.141566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e66085e-ea75-479a-9c9b-ac3cacfc9761 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:33.141876 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e66085e-ea75-479a-9c9b-ac3cacfc9761 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:33.142123 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e66085e-ea75-479a-9c9b-ac3cacfc9761 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:33.161986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-5e66085e-ea75-479a-9c9b-ac3cacfc9761 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:33.163229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-5e66085e-ea75-479a-9c9b-ac3cacfc9761 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:33.175235 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:33.238957 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 505/1009] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:33 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:33.512297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118e5119-e47b-4235-862b-f8b3a8412252 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:33.618429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118e5119-e47b-4235-862b-f8b3a8412252 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:33.652711 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fc7fd4-9987-43d8-832d-efac14adbcfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:33.653345 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fc7fd4-9987-43d8-832d-efac14adbcfa None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:33.653625 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fc7fd4-9987-43d8-832d-efac14adbcfa None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:33.653899 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fc7fd4-9987-43d8-832d-efac14adbcfa None admin] PATH_INFO: `/v3/services/compute` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:33.654474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fc7fd4-9987-43d8-832d-efac14adbcfa None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:33.657093 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fc7fd4-9987-43d8-832d-efac14adbcfa None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:33.665862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-f7fc7fd4-9987-43d8-832d-efac14adbcfa None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 16 03:28:33.665862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:33.665862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:33.665862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:33.665862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:33.665862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:33.665862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:33.665862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:33.665862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 16 03:28:33.665862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:33.665862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:33.665862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:33.665862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:33.665862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:33.665862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:33.665862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:33.665862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:33.666833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:33.666833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:33.666833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:33.666833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:33.666833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:33.666833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:33.666833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:33.666833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:33.666833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:33.666833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:33.666833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:33.666833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:33.666833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:33.666833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:33.666833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:33.666833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:33.668071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:33.668071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:33.668071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:33.668071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:33.668071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:33.668071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:33.668071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:33.668071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:33.668071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:33.668071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:33.668071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:33.668071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:33.668071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 16 03:28:33.668071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:33.668071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 505/1010] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:28:33 2022] GET /identity/v3/services/compute => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:33.673683 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e66085e-ea75-479a-9c9b-ac3cacfc9761 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:33.776241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e66085e-ea75-479a-9c9b-ac3cacfc9761 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:33.814682 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872565b0-1db2-4493-925c-9eac0591d7a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:33.815343 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872565b0-1db2-4493-925c-9eac0591d7a8 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:33.815626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872565b0-1db2-4493-925c-9eac0591d7a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:33.815902 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872565b0-1db2-4493-925c-9eac0591d7a8 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:33.816541 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872565b0-1db2-4493-925c-9eac0591d7a8 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:33.816850 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872565b0-1db2-4493-925c-9eac0591d7a8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:33.818231 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872565b0-1db2-4493-925c-9eac0591d7a8 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:33.828565 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 506/1011] 10.222.0.49 () {56 vars in 1112 bytes} [Fri Sep 16 03:28:33 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:33.829719 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fc7fd4-9987-43d8-832d-efac14adbcfa None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:33.886559 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:33.930657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fc7fd4-9987-43d8-832d-efac14adbcfa None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:33.966066 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c84932b-b70f-4aa5-90ac-aa1e62222277 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:33.966676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c84932b-b70f-4aa5-90ac-aa1e62222277 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:33.966987 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c84932b-b70f-4aa5-90ac-aa1e62222277 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:33.967267 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c84932b-b70f-4aa5-90ac-aa1e62222277 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:33.967894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c84932b-b70f-4aa5-90ac-aa1e62222277 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:33.968199 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c84932b-b70f-4aa5-90ac-aa1e62222277 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:33.969739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c84932b-b70f-4aa5-90ac-aa1e62222277 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:33.974791 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 506/1012] 10.222.0.49 () {56 vars in 1112 bytes} [Fri Sep 16 03:28:33 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:33.980310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872565b0-1db2-4493-925c-9eac0591d7a8 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:33.992835 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:34.082935 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872565b0-1db2-4493-925c-9eac0591d7a8 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:34.116327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78f34aa-49e5-4941-aa1c-dcf4383526d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:34.116968 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78f34aa-49e5-4941-aa1c-dcf4383526d9 None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:34.117253 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78f34aa-49e5-4941-aa1c-dcf4383526d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:34.117529 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78f34aa-49e5-4941-aa1c-dcf4383526d9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:34.118192 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78f34aa-49e5-4941-aa1c-dcf4383526d9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:34.119591 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78f34aa-49e5-4941-aa1c-dcf4383526d9 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:34.196811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 507/1013] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:28:33 2022] POST /identity/v3/endpoints => generated 334 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:28:35.072165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d762386-3f94-4e84-a217-a218dc9b1b81 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:35.072825 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d762386-3f94-4e84-a217-a218dc9b1b81 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:35.072825 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d762386-3f94-4e84-a217-a218dc9b1b81 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:35.074065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 507/1014] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:35.079769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9bd38d-897c-4d9f-9b7f-2493a4f35065 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:35.080176 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9bd38d-897c-4d9f-9b7f-2493a4f35065 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:35.080377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9bd38d-897c-4d9f-9b7f-2493a4f35065 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:35.098084 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-0a9bd38d-897c-4d9f-9b7f-2493a4f35065 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:35.099284 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-0a9bd38d-897c-4d9f-9b7f-2493a4f35065 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:35.157930 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:35.172702 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 508/1015] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:35 2022] POST /identity/v3/auth/tokens => generated 1873 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 03:28:35.306611 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ebefd2-c8fc-40b6-9fa3-be18b28e89d1 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:35.306843 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ebefd2-c8fc-40b6-9fa3-be18b28e89d1 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:35.310012 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ebefd2-c8fc-40b6-9fa3-be18b28e89d1 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:35.310012 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 508/1016] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:35.314264 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa13532e-b96a-4485-9bd7-f3a8674de19b None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:35.314555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa13532e-b96a-4485-9bd7-f3a8674de19b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:35.314832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa13532e-b96a-4485-9bd7-f3a8674de19b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:35.334066 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-fa13532e-b96a-4485-9bd7-f3a8674de19b None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:35.335300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-fa13532e-b96a-4485-9bd7-f3a8674de19b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:35.406887 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 509/1017] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:35 2022] POST /identity/v3/auth/tokens => generated 1873 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:35.442050 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ebefd2-c8fc-40b6-9fa3-be18b28e89d1 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:35.544251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ebefd2-c8fc-40b6-9fa3-be18b28e89d1 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:35.562305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:35.580367 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39008e7e-2c46-46a0-8907-f940c372992e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:35.580367 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39008e7e-2c46-46a0-8907-f940c372992e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:35.580865 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39008e7e-2c46-46a0-8907-f940c372992e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:35.581328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39008e7e-2c46-46a0-8907-f940c372992e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:35.582601 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39008e7e-2c46-46a0-8907-f940c372992e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:35.584226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39008e7e-2c46-46a0-8907-f940c372992e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:35.586298 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 509/1018] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:35.591722 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa13532e-b96a-4485-9bd7-f3a8674de19b None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:35.688921 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39008e7e-2c46-46a0-8907-f940c372992e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:35.696259 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa13532e-b96a-4485-9bd7-f3a8674de19b None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:35.721177 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:35.735658 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61fece0-45a2-4115-9067-2386181c3a66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:35.736471 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61fece0-45a2-4115-9067-2386181c3a66 None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:35.737012 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61fece0-45a2-4115-9067-2386181c3a66 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:35.737136 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61fece0-45a2-4115-9067-2386181c3a66 None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:35.737789 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61fece0-45a2-4115-9067-2386181c3a66 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:35.739298 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61fece0-45a2-4115-9067-2386181c3a66 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:35.794343 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39008e7e-2c46-46a0-8907-f940c372992e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:35.829231 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9765ef82-4373-49c7-95f4-f6eb5cb22f22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:35.829827 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9765ef82-4373-49c7-95f4-f6eb5cb22f22 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:35.830105 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9765ef82-4373-49c7-95f4-f6eb5cb22f22 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:35.830427 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9765ef82-4373-49c7-95f4-f6eb5cb22f22 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:35.831050 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9765ef82-4373-49c7-95f4-f6eb5cb22f22 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:35.832414 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9765ef82-4373-49c7-95f4-f6eb5cb22f22 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:35.852230 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 510/1019] 10.222.0.49 () {60 vars in 1141 bytes} [Fri Sep 16 03:28:35 2022] POST /identity/v3/projects => generated 311 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:28:35.912519 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 510/1020] 10.222.0.49 () {60 vars in 1141 bytes} [Fri Sep 16 03:28:35 2022] POST /identity/v3/services => generated 242 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:28:36.479472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fb84ab-1f98-4b72-8812-08d1addf63b7 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:36.480019 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fb84ab-1f98-4b72-8812-08d1addf63b7 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:36.480019 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fb84ab-1f98-4b72-8812-08d1addf63b7 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:36.481361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 511/1021] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:36.487671 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93dfc56-7458-4be8-a350-f4d5e06b6a34 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:36.488095 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93dfc56-7458-4be8-a350-f4d5e06b6a34 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:36.488220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93dfc56-7458-4be8-a350-f4d5e06b6a34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:36.507173 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-d93dfc56-7458-4be8-a350-f4d5e06b6a34 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:36.508350 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d93dfc56-7458-4be8-a350-f4d5e06b6a34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:36.552403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:36.582487 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 511/1022] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:36 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:36.854407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fb84ab-1f98-4b72-8812-08d1addf63b7 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:36.972883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fb84ab-1f98-4b72-8812-08d1addf63b7 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:37.020161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b32b2d4-1d5c-4b68-a96d-4e816d053d5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:37.021317 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b32b2d4-1d5c-4b68-a96d-4e816d053d5a None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:37.021861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b32b2d4-1d5c-4b68-a96d-4e816d053d5a None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:37.022368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b32b2d4-1d5c-4b68-a96d-4e816d053d5a None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:37.034446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b32b2d4-1d5c-4b68-a96d-4e816d053d5a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:28:37.039650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-3b32b2d4-1d5c-4b68-a96d-4e816d053d5a None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 03:28:37.039650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:37.039650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:37.039650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:37.039650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:37.039650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:37.039650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:37.039650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:37.039650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:37.039650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:37.039650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:37.039650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:37.039650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:28:37.039650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:28:37.039650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:28:37.039650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:28:37.040597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:28:37.040597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:28:37.040597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:28:37.040597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:28:37.040597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:37.040597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:37.040597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:37.040597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:37.040597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:37.040597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:37.040597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:37.040597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:37.040597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:37.040597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:37.040597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:37.040597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:37.041473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:37.041473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:37.041473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:37.041473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:37.041473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:28:37.041473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:28:37.041473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:28:37.041473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:28:37.041473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:28:37.041473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:28:37.041473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 03:28:37.041473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:37.043808 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 512/1023] 10.222.0.49 () {56 vars in 1120 bytes} [Fri Sep 16 03:28:36 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:37.050380 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93dfc56-7458-4be8-a350-f4d5e06b6a34 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:37.150678 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93dfc56-7458-4be8-a350-f4d5e06b6a34 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:37.185285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dc51b4-c46c-4ac4-b213-6359d76e4270 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:37.185898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dc51b4-c46c-4ac4-b213-6359d76e4270 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:37.186175 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dc51b4-c46c-4ac4-b213-6359d76e4270 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:37.186444 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dc51b4-c46c-4ac4-b213-6359d76e4270 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:37.187721 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07dc51b4-c46c-4ac4-b213-6359d76e4270 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:37.188144 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07dc51b4-c46c-4ac4-b213-6359d76e4270 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:37.189566 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07dc51b4-c46c-4ac4-b213-6359d76e4270 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:37.197773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 512/1024] 10.222.0.49 () {56 vars in 1115 bytes} [Fri Sep 16 03:28:37 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:37.203383 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b32b2d4-1d5c-4b68-a96d-4e816d053d5a None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:37.236261 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:37.309540 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b32b2d4-1d5c-4b68-a96d-4e816d053d5a None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:37.347904 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82af2f68-5698-4e88-96c8-46ffc1bf2b98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:37.348543 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82af2f68-5698-4e88-96c8-46ffc1bf2b98 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:37.348826 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82af2f68-5698-4e88-96c8-46ffc1bf2b98 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:37.349101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82af2f68-5698-4e88-96c8-46ffc1bf2b98 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:37.360891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82af2f68-5698-4e88-96c8-46ffc1bf2b98 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:37.362352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82af2f68-5698-4e88-96c8-46ffc1bf2b98 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:37.377228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-82af2f68-5698-4e88-96c8-46ffc1bf2b98 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:37.377228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:37.377228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:37.377228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:37.377228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:37.377228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:37.377228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:37.377228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:37.377228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:37.377228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:37.377228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:37.377228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:37.377228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:37.377228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:37.377228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:37.377228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:37.378174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:37.378174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:37.378174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:37.378174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:37.378174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:37.378174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:37.378174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:37.378174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:37.378174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:37.378174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:37.378174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:37.378174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:37.378174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:37.378174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:37.378174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:37.378174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:37.378174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:37.379164 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:37.379164 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:37.379164 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:37.384581 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 513/1025] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:37.384704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dc51b4-c46c-4ac4-b213-6359d76e4270 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:37.419866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:37.492193 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dc51b4-c46c-4ac4-b213-6359d76e4270 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:37.527589 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a600b0-576f-4050-ae63-6ecdfa5636ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:37.528466 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a600b0-576f-4050-ae63-6ecdfa5636ac None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:37.528925 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a600b0-576f-4050-ae63-6ecdfa5636ac None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:37.529375 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a600b0-576f-4050-ae63-6ecdfa5636ac None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:37.530201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a600b0-576f-4050-ae63-6ecdfa5636ac None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:37.531001 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a600b0-576f-4050-ae63-6ecdfa5636ac None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:37.532545 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a600b0-576f-4050-ae63-6ecdfa5636ac None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:37.544111 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 513/1026] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:37.549982 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82af2f68-5698-4e88-96c8-46ffc1bf2b98 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:37.639169 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:37.655155 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82af2f68-5698-4e88-96c8-46ffc1bf2b98 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:37.692441 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a899a9-f10f-42cc-99f1-f7dffbc6fec5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:37.693084 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a899a9-f10f-42cc-99f1-f7dffbc6fec5 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:37.693366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a899a9-f10f-42cc-99f1-f7dffbc6fec5 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:37.693640 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a899a9-f10f-42cc-99f1-f7dffbc6fec5 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:37.701384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a899a9-f10f-42cc-99f1-f7dffbc6fec5 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:37.702879 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a899a9-f10f-42cc-99f1-f7dffbc6fec5 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:37.714318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-a7a899a9-f10f-42cc-99f1-f7dffbc6fec5 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 16 03:28:37.714318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:37.714318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:37.714318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:37.714318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:37.714318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:37.714318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:37.714318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:37.714318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:37.714318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:37.714318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:37.714318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:37.714318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:28:37.714318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:28:37.714318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:28:37.714318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:28:37.715348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:37.715348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:37.715348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:28:37.715348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:37.715348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:28:37.715348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:37.715348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:37.715348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:37.715348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:37.715348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:37.715348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:37.715348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:37.715348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:37.715348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:37.715348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:37.715348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:37.715348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:37.715348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:37.716387 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:37.716387 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:37.716387 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:28:37.716387 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:28:37.716387 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:28:37.716387 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:28:37.716387 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:28:37.716387 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:28:37.716387 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 16 03:28:37.716387 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:37.719257 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 514/1027] 10.222.0.49 () {56 vars in 1093 bytes} [Fri Sep 16 03:28:37 2022] GET /identity/v3/users/nova => generated 82 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:37.721321 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a600b0-576f-4050-ae63-6ecdfa5636ac None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:37.815991 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:37.823189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a600b0-576f-4050-ae63-6ecdfa5636ac None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:37.857087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5809f3-64ec-4302-bf85-c30d5c3f942a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:37.857814 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5809f3-64ec-4302-bf85-c30d5c3f942a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:37.858100 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5809f3-64ec-4302-bf85-c30d5c3f942a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:37.858383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5809f3-64ec-4302-bf85-c30d5c3f942a None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:37.859658 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5809f3-64ec-4302-bf85-c30d5c3f942a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:37.859965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5809f3-64ec-4302-bf85-c30d5c3f942a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:37.861393 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5809f3-64ec-4302-bf85-c30d5c3f942a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:37.878548 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 514/1028] 10.222.0.49 () {56 vars in 1133 bytes} [Fri Sep 16 03:28:37 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:37.884022 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a899a9-f10f-42cc-99f1-f7dffbc6fec5 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:37.993739 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a899a9-f10f-42cc-99f1-f7dffbc6fec5 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:38.030537 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fe0812-d842-4d17-ac74-e33632abc511 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:38.031817 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fe0812-d842-4d17-ac74-e33632abc511 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:38.032096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fe0812-d842-4d17-ac74-e33632abc511 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:38.032367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fe0812-d842-4d17-ac74-e33632abc511 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:38.046879 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:38.086420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7c8d38-fa0d-4efa-80fe-cf2ab61f38c2 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:38.086691 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7c8d38-fa0d-4efa-80fe-cf2ab61f38c2 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:38.086997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7c8d38-fa0d-4efa-80fe-cf2ab61f38c2 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:38.088319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 515/1029] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:28:38.093125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b93eab0-a3df-4c6b-8a3c-3d9ae85cd5c0 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:38.093437 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b93eab0-a3df-4c6b-8a3c-3d9ae85cd5c0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:38.093707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b93eab0-a3df-4c6b-8a3c-3d9ae85cd5c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:38.105811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9fe0812-d842-4d17-ac74-e33632abc511 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:38.107291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9fe0812-d842-4d17-ac74-e33632abc511 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:38.111416 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7b93eab0-a3df-4c6b-8a3c-3d9ae85cd5c0 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:38.112587 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7b93eab0-a3df-4c6b-8a3c-3d9ae85cd5c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:38.179284 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 516/1030] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:38 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 03:28:38.188392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a23aaf6-be45-4321-932c-ec9427f18193 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:38.188674 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a23aaf6-be45-4321-932c-ec9427f18193 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:38.188945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a23aaf6-be45-4321-932c-ec9427f18193 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:38.190319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 517/1031] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:38 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:38.195764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca9f6d5-1cf9-4d4e-9767-5536ec2c9357 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:38.196177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca9f6d5-1cf9-4d4e-9767-5536ec2c9357 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:38.196461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca9f6d5-1cf9-4d4e-9767-5536ec2c9357 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:38.208958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 515/1032] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:28:37 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 03:28:38.211691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fe0812-d842-4d17-ac74-e33632abc511 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:38.214629 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-4ca9f6d5-1cf9-4d4e-9767-5536ec2c9357 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:38.215860 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4ca9f6d5-1cf9-4d4e-9767-5536ec2c9357 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:38.259468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:38.287343 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 518/1033] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:38 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 03:28:38.318602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fe0812-d842-4d17-ac74-e33632abc511 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:38.353823 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be7d45f-b01c-45a6-a703-bd31cd15adc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:38.354453 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be7d45f-b01c-45a6-a703-bd31cd15adc1 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:38.354733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be7d45f-b01c-45a6-a703-bd31cd15adc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:38.355040 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be7d45f-b01c-45a6-a703-bd31cd15adc1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:38.365055 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be7d45f-b01c-45a6-a703-bd31cd15adc1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:38.366485 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be7d45f-b01c-45a6-a703-bd31cd15adc1 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:38.381037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-2be7d45f-b01c-45a6-a703-bd31cd15adc1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:38.381037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:38.381037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:38.381037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:38.381037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:38.381037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:38.381037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:38.381037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:38.381037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:38.381037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:38.381037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:38.381037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:38.381037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:38.381037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:38.381037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:38.381037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:38.381978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:38.381978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:38.381978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:38.381978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:38.381978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:38.381978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:38.381978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:38.381978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:38.381978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:38.381978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:38.381978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:38.381978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:38.381978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:38.381978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:38.381978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:38.381978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:38.381978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:38.387881 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:38.387881 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:38.387881 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:38.387881 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 516/1034] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:38.388161 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca9f6d5-1cf9-4d4e-9767-5536ec2c9357 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:38.403942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:38.452642 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be7d45f-b01c-45a6-a703-bd31cd15adc1 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:38.495759 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca9f6d5-1cf9-4d4e-9767-5536ec2c9357 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:38.531565 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e756bcc-93a5-4a6e-b570-98a830aa101e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:38.532263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e756bcc-93a5-4a6e-b570-98a830aa101e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:38.532544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e756bcc-93a5-4a6e-b570-98a830aa101e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:38.532817 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e756bcc-93a5-4a6e-b570-98a830aa101e None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:38.533452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e756bcc-93a5-4a6e-b570-98a830aa101e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:38.533755 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e756bcc-93a5-4a6e-b570-98a830aa101e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:38.535571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e756bcc-93a5-4a6e-b570-98a830aa101e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:38.547985 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 519/1035] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:38.553225 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e756bcc-93a5-4a6e-b570-98a830aa101e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:38.556102 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:38.563340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be7d45f-b01c-45a6-a703-bd31cd15adc1 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:38.600618 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2286d185-2d68-4646-b521-3f935ed35683 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:38.601254 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2286d185-2d68-4646-b521-3f935ed35683 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:38.601536 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2286d185-2d68-4646-b521-3f935ed35683 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:38.601814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2286d185-2d68-4646-b521-3f935ed35683 None admin] PATH_INFO: `/v3/services/image` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:38.602396 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2286d185-2d68-4646-b521-3f935ed35683 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:38.603809 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2286d185-2d68-4646-b521-3f935ed35683 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:38.612549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-2286d185-2d68-4646-b521-3f935ed35683 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 16 03:28:38.612549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:38.612549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:38.612549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:38.612549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:38.612549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:38.612549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:38.612549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:38.612549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 16 03:28:38.612549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:38.612549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:38.612549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:38.612549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:38.612549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:38.612549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:38.612549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:38.612549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:38.613774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:38.613774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:38.613774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:38.613774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:38.613774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:38.613774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:38.613774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:38.613774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:38.613774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:38.613774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:38.613774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:38.613774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:38.613774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:38.613774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:38.613774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:38.613774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:38.614845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:38.614845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:38.614845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:38.614845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:38.614845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:38.614845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:38.614845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:38.614845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:38.614845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:38.614845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:38.614845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:38.614845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:38.614845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 16 03:28:38.614845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:38.614845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 517/1036] 10.222.0.49 () {56 vars in 1105 bytes} [Fri Sep 16 03:28:38 2022] GET /identity/v3/services/image => generated 86 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:38.615729 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2286d185-2d68-4646-b521-3f935ed35683 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:38.628785 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:38.658752 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e756bcc-93a5-4a6e-b570-98a830aa101e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:38.692908 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa03cdf-2c4b-4997-8d74-9aedd1c1a570 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:38.693535 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa03cdf-2c4b-4997-8d74-9aedd1c1a570 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:38.693825 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa03cdf-2c4b-4997-8d74-9aedd1c1a570 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:38.694099 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa03cdf-2c4b-4997-8d74-9aedd1c1a570 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:38.700617 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa03cdf-2c4b-4997-8d74-9aedd1c1a570 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:38.702036 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa03cdf-2c4b-4997-8d74-9aedd1c1a570 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:38.712825 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2286d185-2d68-4646-b521-3f935ed35683 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:38.713523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-cfa03cdf-2c4b-4997-8d74-9aedd1c1a570 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:38.713523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:38.713523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:38.713523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:38.713523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:38.713523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:38.713523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:38.713523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:38.713523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:38.713523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:38.713523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:38.713523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:38.713523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:28:38.713523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:28:38.713523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:28:38.714489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:28:38.714489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:38.714489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:38.714489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:38.714489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:38.714489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:38.714489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:38.714489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:38.714489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:38.714489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:38.714489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:38.714489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:38.714489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:38.714489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:38.714489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:38.714489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:38.714489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:38.715493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:28:38.715493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:28:38.715493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:28:38.715493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:28:38.715493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:28:38.715493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:28:38.715493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:38.715493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:38.715493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 520/1037] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:28:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:38.715493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa03cdf-2c4b-4997-8d74-9aedd1c1a570 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:38.749156 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682b3853-d09e-4aba-9657-45d5d1b6d794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:38.750335 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682b3853-d09e-4aba-9657-45d5d1b6d794 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:38.750605 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682b3853-d09e-4aba-9657-45d5d1b6d794 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:38.750873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682b3853-d09e-4aba-9657-45d5d1b6d794 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:38.752006 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682b3853-d09e-4aba-9657-45d5d1b6d794 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:38.753425 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682b3853-d09e-4aba-9657-45d5d1b6d794 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:38.754986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 518/1038] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:38.755736 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682b3853-d09e-4aba-9657-45d5d1b6d794 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:38.783787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:38.816516 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa03cdf-2c4b-4997-8d74-9aedd1c1a570 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:38.845274 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:38.850073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a381e4-91a0-40e7-9a75-1950bb1424e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:38.850672 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a381e4-91a0-40e7-9a75-1950bb1424e5 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:38.850973 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a381e4-91a0-40e7-9a75-1950bb1424e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:38.851315 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a381e4-91a0-40e7-9a75-1950bb1424e5 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:38.851894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a381e4-91a0-40e7-9a75-1950bb1424e5 None admin] RBAC: Adding query filter params (name=image) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:38.852192 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a381e4-91a0-40e7-9a75-1950bb1424e5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:38.853538 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a381e4-91a0-40e7-9a75-1950bb1424e5 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:38.857480 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682b3853-d09e-4aba-9657-45d5d1b6d794 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:38.858229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 521/1039] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:38 2022] GET /identity/v3/services?name=image => generated 122 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:38.859231 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a381e4-91a0-40e7-9a75-1950bb1424e5 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:38.891519 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26352bf-137f-486c-930f-089e93641c1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:38.892121 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26352bf-137f-486c-930f-089e93641c1a None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:38.892399 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26352bf-137f-486c-930f-089e93641c1a None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:38.892669 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26352bf-137f-486c-930f-089e93641c1a None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:38.893347 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26352bf-137f-486c-930f-089e93641c1a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:38.893647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26352bf-137f-486c-930f-089e93641c1a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:38.895116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26352bf-137f-486c-930f-089e93641c1a None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:38.906709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 519/1040] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:28:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:38.907511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26352bf-137f-486c-930f-089e93641c1a None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:38.961405 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a381e4-91a0-40e7-9a75-1950bb1424e5 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:38.993143 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:38.994547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df68f2eb-850c-4009-8537-a48e59dc158f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:38.995402 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df68f2eb-850c-4009-8537-a48e59dc158f None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:38.995689 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df68f2eb-850c-4009-8537-a48e59dc158f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:38.995959 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df68f2eb-850c-4009-8537-a48e59dc158f None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:38.996602 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df68f2eb-850c-4009-8537-a48e59dc158f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:38.998029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df68f2eb-850c-4009-8537-a48e59dc158f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:39.004889 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26352bf-137f-486c-930f-089e93641c1a None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:39.037670 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169a6244-024e-4cc7-9639-edb95c0358dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:39.038201 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169a6244-024e-4cc7-9639-edb95c0358dc None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:39.038477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169a6244-024e-4cc7-9639-edb95c0358dc None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:39.038746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169a6244-024e-4cc7-9639-edb95c0358dc None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:39.039391 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-169a6244-024e-4cc7-9639-edb95c0358dc None admin] RBAC: Adding query filter params (type=image) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:39.039692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-169a6244-024e-4cc7-9639-edb95c0358dc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:39.041033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-169a6244-024e-4cc7-9639-edb95c0358dc None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:39.045600 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 520/1041] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:38 2022] GET /identity/v3/services?type=image => generated 350 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:39.046972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169a6244-024e-4cc7-9639-edb95c0358dc None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:39.118287 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 522/1042] 10.222.0.49 () {60 vars in 1132 bytes} [Fri Sep 16 03:28:38 2022] POST /identity/v3/users => generated 285 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:28:39.118287 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df68f2eb-850c-4009-8537-a48e59dc158f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:39.156598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169a6244-024e-4cc7-9639-edb95c0358dc None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:39.174653 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:39.200489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494af3ef-6dbb-4697-9ad3-ce2d63520903 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:39.201207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494af3ef-6dbb-4697-9ad3-ce2d63520903 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:39.201485 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494af3ef-6dbb-4697-9ad3-ce2d63520903 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:39.201754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494af3ef-6dbb-4697-9ad3-ce2d63520903 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:39.202489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494af3ef-6dbb-4697-9ad3-ce2d63520903 None admin] RBAC: Adding query filter params (role.id=4b248d39ed014678bfc26b6f0acd7134, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=ded1bcd10e62499a9aaa0ece5aa55c80) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:39.203365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494af3ef-6dbb-4697-9ad3-ce2d63520903 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:39.204839 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494af3ef-6dbb-4697-9ad3-ce2d63520903 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:39.210942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 521/1043] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:28:39 2022] GET /identity/v3/role_assignments?user.id=ded1bcd10e62499a9aaa0ece5aa55c80&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=4b248d39ed014678bfc26b6f0acd7134 => generated 259 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:39.237118 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df68f2eb-850c-4009-8537-a48e59dc158f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:39.276204 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45cfe5e-5110-4feb-82e3-9c5352b9051a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:39.276947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45cfe5e-5110-4feb-82e3-9c5352b9051a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:39.277297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45cfe5e-5110-4feb-82e3-9c5352b9051a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:39.277569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45cfe5e-5110-4feb-82e3-9c5352b9051a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:39.278299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45cfe5e-5110-4feb-82e3-9c5352b9051a None admin] RBAC: Adding query filter params (interface=public, service_id=6c2b8e6ab0c04df789ab2e2a303def9b, region_id=RegionOne) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:39.278619 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45cfe5e-5110-4feb-82e3-9c5352b9051a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:39.280083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45cfe5e-5110-4feb-82e3-9c5352b9051a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:39.287939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 523/1044] 10.222.0.49 () {56 vars in 1251 bytes} [Fri Sep 16 03:28:39 2022] GET /identity/v3/endpoints?service_id=6c2b8e6ab0c04df789ab2e2a303def9b&interface=public®ion_id=RegionOne => generated 194 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:39.293633 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494af3ef-6dbb-4697-9ad3-ce2d63520903 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:39.317270 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:39.421649 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494af3ef-6dbb-4697-9ad3-ce2d63520903 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:39.460195 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131483a4-ea80-453b-a8ff-6f6228d21899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:39.460807 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131483a4-ea80-453b-a8ff-6f6228d21899 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:39.461089 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131483a4-ea80-453b-a8ff-6f6228d21899 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:39.461363 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131483a4-ea80-453b-a8ff-6f6228d21899 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:39.461967 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131483a4-ea80-453b-a8ff-6f6228d21899 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:39.463373 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131483a4-ea80-453b-a8ff-6f6228d21899 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:39.478459 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 522/1045] 10.222.0.49 () {56 vars in 1087 bytes} [Fri Sep 16 03:28:39 2022] GET /identity/v3/services => generated 1957 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:28:41.356608 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e411499d-063f-4a3b-993f-106578898689 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:41.357151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e411499d-063f-4a3b-993f-106578898689 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:41.357151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e411499d-063f-4a3b-993f-106578898689 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:41.362929 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 524/1046] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:41.364551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e250f91-7049-4f99-9cbc-068c40070bc9 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:41.364956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e250f91-7049-4f99-9cbc-068c40070bc9 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:41.365128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e250f91-7049-4f99-9cbc-068c40070bc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:41.372835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d435ef43-49d4-4283-8381-9e6cd93bb351 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:41.373117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d435ef43-49d4-4283-8381-9e6cd93bb351 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:41.373390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d435ef43-49d4-4283-8381-9e6cd93bb351 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:41.375500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 525/1047] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:28:41.380878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacf3bbf-2aa1-4dbf-a2d9-a49aac656e85 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:41.381161 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacf3bbf-2aa1-4dbf-a2d9-a49aac656e85 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:41.381434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacf3bbf-2aa1-4dbf-a2d9-a49aac656e85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:41.383818 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-4e250f91-7049-4f99-9cbc-068c40070bc9 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:41.384988 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-4e250f91-7049-4f99-9cbc-068c40070bc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:41.400381 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-aacf3bbf-2aa1-4dbf-a2d9-a49aac656e85 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:41.401637 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-aacf3bbf-2aa1-4dbf-a2d9-a49aac656e85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:41.453606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 523/1048] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:41 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:41.473428 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 526/1049] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:41 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 03:28:41.680242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72766671-28d4-4cee-b3ae-ef7965c300f6 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:41.680435 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72766671-28d4-4cee-b3ae-ef7965c300f6 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:41.680721 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72766671-28d4-4cee-b3ae-ef7965c300f6 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:41.682207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 524/1050] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:41.687502 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7286b8c3-2bd6-48de-b20d-9d1747a08971 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:41.687790 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7286b8c3-2bd6-48de-b20d-9d1747a08971 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:41.688068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7286b8c3-2bd6-48de-b20d-9d1747a08971 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:41.706072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7286b8c3-2bd6-48de-b20d-9d1747a08971 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:41.707286 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7286b8c3-2bd6-48de-b20d-9d1747a08971 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:41.729715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72766671-28d4-4cee-b3ae-ef7965c300f6 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:41.743516 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:41.753181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:41.776832 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 527/1051] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:41 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:41.783114 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7286b8c3-2bd6-48de-b20d-9d1747a08971 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:41.834279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72766671-28d4-4cee-b3ae-ef7965c300f6 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:41.870800 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db34f8b-578c-43e9-ba8b-836f55bd55a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:41.871699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db34f8b-578c-43e9-ba8b-836f55bd55a7 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:41.872166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db34f8b-578c-43e9-ba8b-836f55bd55a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:41.872447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db34f8b-578c-43e9-ba8b-836f55bd55a7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:41.883787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db34f8b-578c-43e9-ba8b-836f55bd55a7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:28:41.887675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7286b8c3-2bd6-48de-b20d-9d1747a08971 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:41.888737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-9db34f8b-578c-43e9-ba8b-836f55bd55a7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:28:41.888737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:41.888737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:41.888737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:41.888737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:41.888737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:41.888737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:41.888737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:41.888737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:41.888737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:41.888737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:41.888737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:41.888737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:28:41.888737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:28:41.888737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:28:41.888737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:28:41.889746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:28:41.889746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:28:41.889746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:28:41.889746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:28:41.889746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:41.889746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:41.889746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:41.889746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:41.889746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:41.889746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:41.889746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:41.889746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:41.889746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:41.889746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:41.889746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:41.889746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:41.891283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:41.891283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:41.891283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:41.891283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:41.891283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:28:41.891283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:28:41.891283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:28:41.891283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:28:41.891283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:28:41.891283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:28:41.891283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:28:41.891283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:41.891283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 525/1052] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:28:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:41.896333 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db34f8b-578c-43e9-ba8b-836f55bd55a7 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:41.927030 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e94fc7a-3ffd-40f1-b040-cf52e5c8bc3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:41.927669 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e94fc7a-3ffd-40f1-b040-cf52e5c8bc3d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:41.927947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e94fc7a-3ffd-40f1-b040-cf52e5c8bc3d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:41.928216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e94fc7a-3ffd-40f1-b040-cf52e5c8bc3d None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:41.940148 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e94fc7a-3ffd-40f1-b040-cf52e5c8bc3d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:28:41.945357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-3e94fc7a-3ffd-40f1-b040-cf52e5c8bc3d None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 03:28:41.945357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:41.945357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:41.945357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:41.945357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:41.945357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:41.945357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:41.945357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:41.945357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:41.945357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:41.945357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:41.945357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:41.945357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:28:41.945357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:28:41.945357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:28:41.945357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:28:41.946319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:28:41.946319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:28:41.946319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:28:41.946319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:28:41.946319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:41.946319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:41.946319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:41.946319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:41.946319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:41.946319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:41.946319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:41.946319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:41.946319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:41.946319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:41.946319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:41.946319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:41.947378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:41.947378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:41.947378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:41.947378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:41.947378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:28:41.947378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:28:41.947378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:28:41.947378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:28:41.947378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:28:41.947378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:28:41.947378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 03:28:41.947378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:41.947378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 528/1053] 10.222.0.49 () {56 vars in 1120 bytes} [Fri Sep 16 03:28:41 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:41.952733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e94fc7a-3ffd-40f1-b040-cf52e5c8bc3d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:41.969448 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:41.990780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:42.000469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db34f8b-578c-43e9-ba8b-836f55bd55a7 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:42.034159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5927d602-57e0-4324-bf75-4c280883841a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:42.034761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5927d602-57e0-4324-bf75-4c280883841a None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:42.040994 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5927d602-57e0-4324-bf75-4c280883841a None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:42.040994 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5927d602-57e0-4324-bf75-4c280883841a None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:42.040994 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5927d602-57e0-4324-bf75-4c280883841a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:42.040994 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5927d602-57e0-4324-bf75-4c280883841a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:42.040994 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5927d602-57e0-4324-bf75-4c280883841a None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:42.045960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 526/1054] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:28:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:42.052407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5927d602-57e0-4324-bf75-4c280883841a None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:42.056311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e94fc7a-3ffd-40f1-b040-cf52e5c8bc3d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:42.090032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306724e9-bcec-4768-a7b9-1e6caac22507 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:42.090639 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306724e9-bcec-4768-a7b9-1e6caac22507 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:42.090940 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306724e9-bcec-4768-a7b9-1e6caac22507 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:42.091222 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306724e9-bcec-4768-a7b9-1e6caac22507 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:42.091910 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306724e9-bcec-4768-a7b9-1e6caac22507 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:42.092233 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306724e9-bcec-4768-a7b9-1e6caac22507 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:42.095406 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306724e9-bcec-4768-a7b9-1e6caac22507 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:42.109855 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 529/1055] 10.222.0.49 () {56 vars in 1115 bytes} [Fri Sep 16 03:28:41 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:42.109855 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306724e9-bcec-4768-a7b9-1e6caac22507 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:42.164882 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5927d602-57e0-4324-bf75-4c280883841a None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:42.194495 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:42.199649 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e292a95-6b32-43c4-8f2c-023261c54b41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:42.200277 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e292a95-6b32-43c4-8f2c-023261c54b41 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:42.200557 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e292a95-6b32-43c4-8f2c-023261c54b41 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:42.200828 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e292a95-6b32-43c4-8f2c-023261c54b41 None admin] PATH_INFO: `/v3/users/3d5a41fdfbce4585953397195fa25c49` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:42.208558 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:42.212003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306724e9-bcec-4768-a7b9-1e6caac22507 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:42.218501 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e292a95-6b32-43c4-8f2c-023261c54b41 None admin] RBAC: Authorizing `identity:get_user(user_id=3d5a41fdfbce4585953397195fa25c49)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:42.218501 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e292a95-6b32-43c4-8f2c-023261c54b41 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:42.218501 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 527/1056] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:28:42 2022] GET /identity/v3/users/3d5a41fdfbce4585953397195fa25c49 => generated 285 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:42.218501 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e292a95-6b32-43c4-8f2c-023261c54b41 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:42.249511 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b66b0c2-0450-4c95-925c-034063392c5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:42.251869 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b66b0c2-0450-4c95-925c-034063392c5a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:42.251869 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b66b0c2-0450-4c95-925c-034063392c5a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:42.251869 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b66b0c2-0450-4c95-925c-034063392c5a None admin] PATH_INFO: `/v3/services/image` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:42.252168 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b66b0c2-0450-4c95-925c-034063392c5a None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:42.253410 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b66b0c2-0450-4c95-925c-034063392c5a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:42.261960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-1b66b0c2-0450-4c95-925c-034063392c5a None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 16 03:28:42.261960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:42.261960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:28:42.261960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:28:42.261960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:28:42.261960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:28:42.261960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:28:42.261960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:42.261960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 16 03:28:42.261960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:42.261960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:28:42.261960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:42.261960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:42.261960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:42.261960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:42.261960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:42.261960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:42.263297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:42.263297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:42.263297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:42.263297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:42.263297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:42.263297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:42.263297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:28:42.263297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:28:42.263297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:28:42.263297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:28:42.263297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:42.263297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:42.263297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:42.263297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:42.263297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:42.263297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:42.264215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:42.264215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:42.264215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:42.264215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:42.264215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:42.264215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:42.264215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:42.264215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:42.264215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:42.264215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:42.264215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:28:42.264215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:28:42.264215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 16 03:28:42.264215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:42.264215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 530/1057] 10.222.0.49 () {56 vars in 1105 bytes} [Fri Sep 16 03:28:42 2022] GET /identity/v3/services/image => generated 86 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:42.265086 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b66b0c2-0450-4c95-925c-034063392c5a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:42.325004 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e292a95-6b32-43c4-8f2c-023261c54b41 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:42.366560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b66b0c2-0450-4c95-925c-034063392c5a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:42.398840 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf1d77b-e503-4f32-af28-2e805836c2c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:42.399466 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf1d77b-e503-4f32-af28-2e805836c2c7 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:42.399748 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf1d77b-e503-4f32-af28-2e805836c2c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:42.400017 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf1d77b-e503-4f32-af28-2e805836c2c7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:42.429094 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:42.439687 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60aae320-002c-456d-869e-288c4226ba4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:42.440331 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60aae320-002c-456d-869e-288c4226ba4f None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:42.440617 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60aae320-002c-456d-869e-288c4226ba4f None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:42.440891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60aae320-002c-456d-869e-288c4226ba4f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:42.451063 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60aae320-002c-456d-869e-288c4226ba4f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:42.452450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60aae320-002c-456d-869e-288c4226ba4f None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:42.467159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-60aae320-002c-456d-869e-288c4226ba4f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:42.467159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:42.467159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:42.467159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:42.467159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:42.467159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:42.467159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:42.467159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:42.467159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:42.467159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:42.467159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:42.467159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:42.467159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:42.467159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:42.467159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:42.467159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:42.473756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:42.473756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:42.473756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:42.473756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:42.473756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:42.473756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:42.473756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:42.473756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:42.473756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:42.473756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:42.473756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:42.473756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:42.473756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:42.473756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:42.473756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:42.473756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:42.473756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:42.474664 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:42.474664 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:42.474664 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:42.474845 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf1d77b-e503-4f32-af28-2e805836c2c7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:42.474957 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 528/1058] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:42.474957 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60aae320-002c-456d-869e-288c4226ba4f None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:42.475103 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf1d77b-e503-4f32-af28-2e805836c2c7 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:42.517876 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:42.574704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60aae320-002c-456d-869e-288c4226ba4f None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:42.575602 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 531/1059] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:28:42 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 03:28:42.576442 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf1d77b-e503-4f32-af28-2e805836c2c7 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:42.613654 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6259cc35-f387-45f3-9015-c30579b8e443 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:42.614256 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6259cc35-f387-45f3-9015-c30579b8e443 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:42.614532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6259cc35-f387-45f3-9015-c30579b8e443 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:42.614800 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6259cc35-f387-45f3-9015-c30579b8e443 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:42.615521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6259cc35-f387-45f3-9015-c30579b8e443 None admin] RBAC: Adding query filter params (name=image) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:42.615820 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6259cc35-f387-45f3-9015-c30579b8e443 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:42.617180 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6259cc35-f387-45f3-9015-c30579b8e443 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:42.621929 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 529/1060] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:42 2022] GET /identity/v3/services?name=image => generated 122 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:42.622688 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6259cc35-f387-45f3-9015-c30579b8e443 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:42.639609 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:42.678032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf1d77b-e503-4f32-af28-2e805836c2c7 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:42.711888 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b3e768-b707-4c3b-a294-7873a612df6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:42.712423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b3e768-b707-4c3b-a294-7873a612df6a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:42.712703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b3e768-b707-4c3b-a294-7873a612df6a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:42.712976 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b3e768-b707-4c3b-a294-7873a612df6a None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:42.713621 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b3e768-b707-4c3b-a294-7873a612df6a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:42.713927 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b3e768-b707-4c3b-a294-7873a612df6a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:42.715324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b3e768-b707-4c3b-a294-7873a612df6a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:42.720779 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6259cc35-f387-45f3-9015-c30579b8e443 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:42.727516 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 532/1061] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:42.727652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:42.728296 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b3e768-b707-4c3b-a294-7873a612df6a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:42.754624 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532e2f3b-5215-46a3-9eba-8e73d148ed04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:42.755373 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532e2f3b-5215-46a3-9eba-8e73d148ed04 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:42.755657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532e2f3b-5215-46a3-9eba-8e73d148ed04 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:42.755930 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532e2f3b-5215-46a3-9eba-8e73d148ed04 None admin] PATH_INFO: `/v3/projects/b95d42ffdb2b4a668f48401c2db87743` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:42.757041 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-532e2f3b-5215-46a3-9eba-8e73d148ed04 None admin] RBAC: Authorizing `identity:get_project(project_id=b95d42ffdb2b4a668f48401c2db87743)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:42.758461 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-532e2f3b-5215-46a3-9eba-8e73d148ed04 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:42.765696 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 530/1062] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:28:42 2022] GET /identity/v3/projects/b95d42ffdb2b4a668f48401c2db87743 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:42.765696 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532e2f3b-5215-46a3-9eba-8e73d148ed04 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:42.834223 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b3e768-b707-4c3b-a294-7873a612df6a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:42.856006 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:42.860697 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532e2f3b-5215-46a3-9eba-8e73d148ed04 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:42.867722 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4de3c0-2069-4489-b126-ae9f4d13e7ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:42.868325 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4de3c0-2069-4489-b126-ae9f4d13e7ae None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:42.869795 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4de3c0-2069-4489-b126-ae9f4d13e7ae None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:42.869795 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4de3c0-2069-4489-b126-ae9f4d13e7ae None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:42.869795 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4de3c0-2069-4489-b126-ae9f4d13e7ae None admin] RBAC: Adding query filter params (type=image) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:42.869795 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4de3c0-2069-4489-b126-ae9f4d13e7ae None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:42.871173 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4de3c0-2069-4489-b126-ae9f4d13e7ae None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:42.876419 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 533/1063] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:42 2022] GET /identity/v3/services?type=image => generated 350 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:42.877185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4de3c0-2069-4489-b126-ae9f4d13e7ae None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:42.896279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2761048-2b67-449e-aee1-74f2a18a5bb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:42.896909 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2761048-2b67-449e-aee1-74f2a18a5bb4 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:42.897187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2761048-2b67-449e-aee1-74f2a18a5bb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:42.897497 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2761048-2b67-449e-aee1-74f2a18a5bb4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:42.904917 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2761048-2b67-449e-aee1-74f2a18a5bb4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:42.906679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2761048-2b67-449e-aee1-74f2a18a5bb4 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:42.917728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-b2761048-2b67-449e-aee1-74f2a18a5bb4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 16 03:28:42.917728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:42.917728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:42.917728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:42.917728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:42.917728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:42.917728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:42.917728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:42.917728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:42.917728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:42.917728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:42.917728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:42.917728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:28:42.917728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:28:42.917728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:28:42.917728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:28:42.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:42.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:42.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:28:42.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:42.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:28:42.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:42.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:42.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:42.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:42.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:42.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:42.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:42.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:42.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:42.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:42.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:42.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:42.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:42.924361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:42.924361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:42.924361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:28:42.924361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:28:42.924361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:28:42.924361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:28:42.924361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:28:42.924361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:28:42.924361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 16 03:28:42.924361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:42.924361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 531/1064] 10.222.0.49 () {56 vars in 1093 bytes} [Fri Sep 16 03:28:42 2022] GET /identity/v3/users/nova => generated 82 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:42.924361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2761048-2b67-449e-aee1-74f2a18a5bb4 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:42.956864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:42.985995 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4de3c0-2069-4489-b126-ae9f4d13e7ae None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:43.020592 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e048d63-0c7c-4dc4-adc5-fa063aabc45b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:43.021194 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e048d63-0c7c-4dc4-adc5-fa063aabc45b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:43.021470 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e048d63-0c7c-4dc4-adc5-fa063aabc45b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:43.021737 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e048d63-0c7c-4dc4-adc5-fa063aabc45b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:43.022446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e048d63-0c7c-4dc4-adc5-fa063aabc45b None admin] RBAC: Adding query filter params (role.id=89c4b3a971224d468d33d2dc411230a2, scope.project.id=b95d42ffdb2b4a668f48401c2db87743, user.id=3d5a41fdfbce4585953397195fa25c49) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:43.022747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e048d63-0c7c-4dc4-adc5-fa063aabc45b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:43.024182 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e048d63-0c7c-4dc4-adc5-fa063aabc45b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:43.026734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2761048-2b67-449e-aee1-74f2a18a5bb4 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:43.029319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 534/1065] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:28:42 2022] GET /identity/v3/role_assignments?user.id=3d5a41fdfbce4585953397195fa25c49&scope.project.id=b95d42ffdb2b4a668f48401c2db87743&role.id=89c4b3a971224d468d33d2dc411230a2 => generated 259 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:43.031975 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e048d63-0c7c-4dc4-adc5-fa063aabc45b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:43.068069 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b40829-50c4-40ef-998a-42975fe71df5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:43.068792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b40829-50c4-40ef-998a-42975fe71df5 None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:43.069139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b40829-50c4-40ef-998a-42975fe71df5 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:43.069414 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b40829-50c4-40ef-998a-42975fe71df5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:43.070092 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b40829-50c4-40ef-998a-42975fe71df5 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:43.071761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b40829-50c4-40ef-998a-42975fe71df5 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:43.088058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:43.142283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e048d63-0c7c-4dc4-adc5-fa063aabc45b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:43.155206 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 532/1066] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:28:42 2022] POST /identity/v3/endpoints => generated 327 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:28:43.198243 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be721612-dbf3-4a1c-89a0-cb2ed0bc8229 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:43.198859 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be721612-dbf3-4a1c-89a0-cb2ed0bc8229 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:43.199169 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be721612-dbf3-4a1c-89a0-cb2ed0bc8229 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:43.199448 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be721612-dbf3-4a1c-89a0-cb2ed0bc8229 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:43.200220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be721612-dbf3-4a1c-89a0-cb2ed0bc8229 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:43.200527 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be721612-dbf3-4a1c-89a0-cb2ed0bc8229 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:43.202015 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be721612-dbf3-4a1c-89a0-cb2ed0bc8229 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:43.224828 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 535/1067] 10.222.0.49 () {56 vars in 1133 bytes} [Fri Sep 16 03:28:43 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:43.231088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b40829-50c4-40ef-998a-42975fe71df5 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:43.350026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:43.361370 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b40829-50c4-40ef-998a-42975fe71df5 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:43.396793 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08821015-e645-4a12-8483-13eb1822a2f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:43.397469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08821015-e645-4a12-8483-13eb1822a2f6 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:43.397746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08821015-e645-4a12-8483-13eb1822a2f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:43.398017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08821015-e645-4a12-8483-13eb1822a2f6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:43.409220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08821015-e645-4a12-8483-13eb1822a2f6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:43.410661 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08821015-e645-4a12-8483-13eb1822a2f6 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:43.425992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-08821015-e645-4a12-8483-13eb1822a2f6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:43.425992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:43.425992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:43.425992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:43.425992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:43.425992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:43.425992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:43.425992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:43.425992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:43.425992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:43.425992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:43.425992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:43.425992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:43.425992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:43.425992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:43.425992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:43.426986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:43.426986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:43.426986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:43.426986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:43.426986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:43.426986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:43.426986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:43.426986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:43.426986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:43.426986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:43.426986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:43.426986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:43.426986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:43.426986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:43.426986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:43.426986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:43.426986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:43.427943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:43.427943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:43.427943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:43.427943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 533/1068] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:43.432751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be721612-dbf3-4a1c-89a0-cb2ed0bc8229 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:43.535175 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be721612-dbf3-4a1c-89a0-cb2ed0bc8229 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:43.537512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:43.570049 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b2a2fd-9a52-42a7-be69-e317db3879d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:43.570666 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b2a2fd-9a52-42a7-be69-e317db3879d1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:43.570967 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b2a2fd-9a52-42a7-be69-e317db3879d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:43.571244 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b2a2fd-9a52-42a7-be69-e317db3879d1 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:43.571876 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b2a2fd-9a52-42a7-be69-e317db3879d1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:43.572175 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b2a2fd-9a52-42a7-be69-e317db3879d1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:43.573524 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b2a2fd-9a52-42a7-be69-e317db3879d1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:43.590125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 536/1069] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:43.591530 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08821015-e645-4a12-8483-13eb1822a2f6 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:43.691692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08821015-e645-4a12-8483-13eb1822a2f6 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:43.726176 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9550f4e6-b8a5-4226-8adb-c3324ce3df0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:43.726806 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9550f4e6-b8a5-4226-8adb-c3324ce3df0e None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:43.727108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9550f4e6-b8a5-4226-8adb-c3324ce3df0e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:43.727381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9550f4e6-b8a5-4226-8adb-c3324ce3df0e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:43.734175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9550f4e6-b8a5-4226-8adb-c3324ce3df0e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:43.735646 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9550f4e6-b8a5-4226-8adb-c3324ce3df0e None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:43.747046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-9550f4e6-b8a5-4226-8adb-c3324ce3df0e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:43.747046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:43.747046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:43.747046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:43.747046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:43.747046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:43.747046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:43.747046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:43.747046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:43.747046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:43.747046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:43.747046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:43.747046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:28:43.747046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:28:43.747046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:28:43.748008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:28:43.748008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:43.748008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:43.748008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:43.748008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:43.748008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:43.748008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:43.748008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:43.748008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:43.748008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:43.748008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:43.748008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:43.748008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:43.748008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:43.748008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:43.748008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:43.748008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:43.748977 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:28:43.748977 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:28:43.748977 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:28:43.748977 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:28:43.748977 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:28:43.748977 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:28:43.748977 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:43.748977 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:43.748977 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 534/1070] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:28:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:43.755666 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b2a2fd-9a52-42a7-be69-e317db3879d1 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:43.859793 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b2a2fd-9a52-42a7-be69-e317db3879d1 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:43.893886 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc3f3e0-465c-4dcf-bcc0-6840b5f5233a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:43.894505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc3f3e0-465c-4dcf-bcc0-6840b5f5233a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:43.894789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc3f3e0-465c-4dcf-bcc0-6840b5f5233a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:43.895100 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc3f3e0-465c-4dcf-bcc0-6840b5f5233a None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:43.895791 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc3f3e0-465c-4dcf-bcc0-6840b5f5233a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:43.896101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc3f3e0-465c-4dcf-bcc0-6840b5f5233a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:43.897572 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc3f3e0-465c-4dcf-bcc0-6840b5f5233a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:43.916121 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 537/1071] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:28:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:43.917686 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9550f4e6-b8a5-4226-8adb-c3324ce3df0e None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:43.920987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:44.019816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9550f4e6-b8a5-4226-8adb-c3324ce3df0e None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:44.052717 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cbe4343-a541-46a6-a8a2-dc94982f59d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:44.053365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbe4343-a541-46a6-a8a2-dc94982f59d0 None admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:44.053643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbe4343-a541-46a6-a8a2-dc94982f59d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:44.053916 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbe4343-a541-46a6-a8a2-dc94982f59d0 None admin] PATH_INFO: `/v3/projects/0bb7350474f349d6aee197ff35536167/users/ded1bcd10e62499a9aaa0ece5aa55c80/roles/4b248d39ed014678bfc26b6f0acd7134` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:44.056029 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cbe4343-a541-46a6-a8a2-dc94982f59d0 None admin] RBAC: Authorizing `identity:create_grant(project_id=0bb7350474f349d6aee197ff35536167, user_id=ded1bcd10e62499a9aaa0ece5aa55c80, role_id=4b248d39ed014678bfc26b6f0acd7134)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:44.057454 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cbe4343-a541-46a6-a8a2-dc94982f59d0 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:44.122402 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 535/1072] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:28:43 2022] PUT /identity/v3/projects/0bb7350474f349d6aee197ff35536167/users/ded1bcd10e62499a9aaa0ece5aa55c80/roles/4b248d39ed014678bfc26b6f0acd7134 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:28:45.199956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a75284f-1d8f-4e28-8af7-13d20fa1016d None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:45.200482 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a75284f-1d8f-4e28-8af7-13d20fa1016d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:45.200482 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a75284f-1d8f-4e28-8af7-13d20fa1016d None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:45.202002 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 538/1073] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:45.207730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d591db-41d2-47bc-97ff-7de641808b57 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:45.208150 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d591db-41d2-47bc-97ff-7de641808b57 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:45.208316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d591db-41d2-47bc-97ff-7de641808b57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:45.226428 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-f3d591db-41d2-47bc-97ff-7de641808b57 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:45.227657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-f3d591db-41d2-47bc-97ff-7de641808b57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:45.285049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:45.295797 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 536/1074] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:45.568531 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a75284f-1d8f-4e28-8af7-13d20fa1016d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:45.596222 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:45.676567 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a75284f-1d8f-4e28-8af7-13d20fa1016d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:45.710367 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c2c573-cefd-4b87-bd79-605e4c03f2f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:45.711036 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c2c573-cefd-4b87-bd79-605e4c03f2f8 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:45.711313 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c2c573-cefd-4b87-bd79-605e4c03f2f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:45.711585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c2c573-cefd-4b87-bd79-605e4c03f2f8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:45.712324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4ed6c5-baa8-4e19-9425-87894dc5905d None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:45.712615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4ed6c5-baa8-4e19-9425-87894dc5905d None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:45.712887 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4ed6c5-baa8-4e19-9425-87894dc5905d None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:45.714233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 537/1075] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:45.720279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2624499-982a-4d03-a918-9739efc646df None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:45.720568 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2624499-982a-4d03-a918-9739efc646df None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:45.720847 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2624499-982a-4d03-a918-9739efc646df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:45.723697 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c2c573-cefd-4b87-bd79-605e4c03f2f8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:28:45.728420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-76c2c573-cefd-4b87-bd79-605e4c03f2f8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:28:45.728420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:45.728420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:45.728420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:45.728420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:45.728420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:45.728420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:45.728420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:45.728420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:45.728420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:45.728420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:45.728420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:45.728420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:28:45.728420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:28:45.728420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:28:45.728420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:28:45.729389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:28:45.729389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:28:45.729389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:28:45.729389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:28:45.729389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:45.729389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:45.729389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:45.729389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:45.729389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:45.729389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:45.729389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:45.729389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:45.729389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:45.729389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:45.729389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:45.729389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:45.730245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:45.730245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:45.730245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:45.730245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:45.730245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:28:45.730245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:28:45.730245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:28:45.730245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:28:45.730245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:28:45.730245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:28:45.730245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:28:45.730245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:45.730245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 539/1076] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:28:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:45.735626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c2c573-cefd-4b87-bd79-605e4c03f2f8 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:45.742427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-a2624499-982a-4d03-a918-9739efc646df None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:45.743633 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a2624499-982a-4d03-a918-9739efc646df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:45.826305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:45.844080 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c2c573-cefd-4b87-bd79-605e4c03f2f8 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:45.844962 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 538/1077] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 03:28:45.878229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaaea123-b670-424e-8584-0ba3c1a75685 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:45.879349 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaea123-b670-424e-8584-0ba3c1a75685 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:45.879845 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaea123-b670-424e-8584-0ba3c1a75685 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:45.880313 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaea123-b670-424e-8584-0ba3c1a75685 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:45.881154 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaaea123-b670-424e-8584-0ba3c1a75685 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:45.881675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaaea123-b670-424e-8584-0ba3c1a75685 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:45.883327 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaaea123-b670-424e-8584-0ba3c1a75685 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:45.896699 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 540/1078] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:28:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:45.897808 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2624499-982a-4d03-a918-9739efc646df None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:45.997912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2624499-982a-4d03-a918-9739efc646df None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:46.022167 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:46.034071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e699a58-cb1a-4268-81d8-937b1427c3be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:46.034790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e699a58-cb1a-4268-81d8-937b1427c3be None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:46.035097 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e699a58-cb1a-4268-81d8-937b1427c3be None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:46.035370 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e699a58-cb1a-4268-81d8-937b1427c3be None admin] PATH_INFO: `/v3/users/3d5a41fdfbce4585953397195fa25c49` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:46.036487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e699a58-cb1a-4268-81d8-937b1427c3be None admin] RBAC: Authorizing `identity:get_user(user_id=3d5a41fdfbce4585953397195fa25c49)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:46.037915 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e699a58-cb1a-4268-81d8-937b1427c3be None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:46.044233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 539/1079] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:28:45 2022] GET /identity/v3/users/3d5a41fdfbce4585953397195fa25c49 => generated 285 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:46.045235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaaea123-b670-424e-8584-0ba3c1a75685 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:46.127295 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e699a58-cb1a-4268-81d8-937b1427c3be None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:46.157155 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaaea123-b670-424e-8584-0ba3c1a75685 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:46.194404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72240ac0-3797-4392-b16c-5faf2d21bc30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:46.195092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72240ac0-3797-4392-b16c-5faf2d21bc30 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:46.195379 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72240ac0-3797-4392-b16c-5faf2d21bc30 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:46.195658 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72240ac0-3797-4392-b16c-5faf2d21bc30 None admin] PATH_INFO: `/v3/projects/b95d42ffdb2b4a668f48401c2db87743` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:46.196780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72240ac0-3797-4392-b16c-5faf2d21bc30 None admin] RBAC: Authorizing `identity:get_project(project_id=b95d42ffdb2b4a668f48401c2db87743)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:46.198872 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72240ac0-3797-4392-b16c-5faf2d21bc30 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:46.206461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 541/1080] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:28:46 2022] GET /identity/v3/projects/b95d42ffdb2b4a668f48401c2db87743 => generated 311 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:46.207173 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72240ac0-3797-4392-b16c-5faf2d21bc30 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:46.224101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:46.232978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e699a58-cb1a-4268-81d8-937b1427c3be None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:46.269203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe03ef3-29c1-4243-a52c-bab0debaac32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:46.270179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe03ef3-29c1-4243-a52c-bab0debaac32 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:46.270622 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe03ef3-29c1-4243-a52c-bab0debaac32 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:46.270898 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe03ef3-29c1-4243-a52c-bab0debaac32 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:46.325210 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72240ac0-3797-4392-b16c-5faf2d21bc30 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:46.335246 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:46.347340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe03ef3-29c1-4243-a52c-bab0debaac32 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:46.348875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe03ef3-29c1-4243-a52c-bab0debaac32 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:46.360600 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2ee2c5-97b2-4394-947e-9827d2ef2c88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:46.361257 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2ee2c5-97b2-4394-947e-9827d2ef2c88 None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:46.361542 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2ee2c5-97b2-4394-947e-9827d2ef2c88 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:46.361885 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2ee2c5-97b2-4394-947e-9827d2ef2c88 None admin] PATH_INFO: `/v3/projects/b95d42ffdb2b4a668f48401c2db87743/users/3d5a41fdfbce4585953397195fa25c49/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:46.363817 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2ee2c5-97b2-4394-947e-9827d2ef2c88 None admin] RBAC: Authorizing `identity:create_grant(project_id=b95d42ffdb2b4a668f48401c2db87743, user_id=3d5a41fdfbce4585953397195fa25c49, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:46.365239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2ee2c5-97b2-4394-947e-9827d2ef2c88 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:46.426113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 542/1081] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:28:46 2022] PUT /identity/v3/projects/b95d42ffdb2b4a668f48401c2db87743/users/3d5a41fdfbce4585953397195fa25c49/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:28:46.470936 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 540/1082] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:28:46 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 03:28:46.474550 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2ee2c5-97b2-4394-947e-9827d2ef2c88 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:46.561243 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:46.588179 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2ee2c5-97b2-4394-947e-9827d2ef2c88 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:46.628222 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6191178-d86b-438f-bc48-ec397621c9b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:46.629141 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6191178-d86b-438f-bc48-ec397621c9b1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:46.629591 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6191178-d86b-438f-bc48-ec397621c9b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:46.630369 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6191178-d86b-438f-bc48-ec397621c9b1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:46.641741 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6191178-d86b-438f-bc48-ec397621c9b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:46.643221 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6191178-d86b-438f-bc48-ec397621c9b1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:46.650582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588e658b-8f92-4489-979f-22e04cdec1c2 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:46.650876 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588e658b-8f92-4489-979f-22e04cdec1c2 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:46.651188 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588e658b-8f92-4489-979f-22e04cdec1c2 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:46.652705 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 541/1083] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:46.657865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55042c12-4848-4ba7-bfb0-6ef3ad6bf1b8 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:46.658155 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55042c12-4848-4ba7-bfb0-6ef3ad6bf1b8 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:46.658434 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55042c12-4848-4ba7-bfb0-6ef3ad6bf1b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:46.658544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-c6191178-d86b-438f-bc48-ec397621c9b1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:46.658544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:46.658544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:46.658544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:46.658544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:46.658544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:46.658544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:46.658544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:46.658544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:46.658544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:46.658544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:46.658544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:46.658544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:46.658544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:46.658544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:46.658544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:46.659622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:46.659622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:46.659622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:46.659622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:46.659622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:46.659622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:46.659622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:46.659622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:46.659622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:46.659622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:46.659622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:46.659622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:46.659622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:46.659622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:46.659622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:46.659622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:46.659622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:46.660938 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:46.660938 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:46.660938 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:46.660938 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 543/1084] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:46.665145 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6191178-d86b-438f-bc48-ec397621c9b1 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:46.678959 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-55042c12-4848-4ba7-bfb0-6ef3ad6bf1b8 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:46.680067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-55042c12-4848-4ba7-bfb0-6ef3ad6bf1b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:46.753011 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 542/1085] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:46 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 03:28:46.774033 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6191178-d86b-438f-bc48-ec397621c9b1 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:46.803613 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:46.812930 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71048585-1fc7-48ec-99d4-a8a18bab6a4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:46.814306 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71048585-1fc7-48ec-99d4-a8a18bab6a4a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:46.814306 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71048585-1fc7-48ec-99d4-a8a18bab6a4a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:46.814306 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71048585-1fc7-48ec-99d4-a8a18bab6a4a None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:46.814735 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71048585-1fc7-48ec-99d4-a8a18bab6a4a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:46.815162 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71048585-1fc7-48ec-99d4-a8a18bab6a4a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:46.816550 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71048585-1fc7-48ec-99d4-a8a18bab6a4a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:46.829291 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 544/1086] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:47.036056 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55042c12-4848-4ba7-bfb0-6ef3ad6bf1b8 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:47.058555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:47.146210 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55042c12-4848-4ba7-bfb0-6ef3ad6bf1b8 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:47.180632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4e4feb-e454-4710-b9d9-aec80c82e93e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:47.181266 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4e4feb-e454-4710-b9d9-aec80c82e93e None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:47.181545 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4e4feb-e454-4710-b9d9-aec80c82e93e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:47.181817 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4e4feb-e454-4710-b9d9-aec80c82e93e None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:47.191597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4e4feb-e454-4710-b9d9-aec80c82e93e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:28:47.196307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-1c4e4feb-e454-4710-b9d9-aec80c82e93e None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 03:28:47.196307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:47.196307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:47.196307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:47.196307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:47.196307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:47.196307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:47.196307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:47.196307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:47.196307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:47.196307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:47.196307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:47.196307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:28:47.196307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:28:47.196307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:28:47.196307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:28:47.197450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:28:47.197450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:28:47.197450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:28:47.197450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:28:47.197450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:47.197450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:47.197450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:47.197450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:47.197450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:47.197450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:47.197450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:47.197450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:47.197450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:47.197450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:47.197450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:47.197450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:47.198340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:47.198340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:47.198340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:47.198340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:47.198340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:28:47.198340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:28:47.198340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:28:47.198340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:28:47.198340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:28:47.198340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:28:47.198340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 03:28:47.198340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:47.198340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 543/1087] 10.222.0.49 () {56 vars in 1120 bytes} [Fri Sep 16 03:28:47 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:47.203588 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71048585-1fc7-48ec-99d4-a8a18bab6a4a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:47.300064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71048585-1fc7-48ec-99d4-a8a18bab6a4a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:47.338053 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3547ff-aa0f-4986-b120-47e573c01888 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:47.338662 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3547ff-aa0f-4986-b120-47e573c01888 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:47.338974 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3547ff-aa0f-4986-b120-47e573c01888 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:47.339246 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3547ff-aa0f-4986-b120-47e573c01888 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:47.339895 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da3547ff-aa0f-4986-b120-47e573c01888 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:47.340191 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da3547ff-aa0f-4986-b120-47e573c01888 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:47.341529 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da3547ff-aa0f-4986-b120-47e573c01888 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:47.349884 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 545/1088] 10.222.0.49 () {56 vars in 1115 bytes} [Fri Sep 16 03:28:47 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:47.355661 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4e4feb-e454-4710-b9d9-aec80c82e93e None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:47.437842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:47.456614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4e4feb-e454-4710-b9d9-aec80c82e93e None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:47.490363 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5943b12c-6dad-4fc6-85c8-e4e3043680d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:47.491027 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5943b12c-6dad-4fc6-85c8-e4e3043680d2 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:47.491311 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5943b12c-6dad-4fc6-85c8-e4e3043680d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:47.491582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5943b12c-6dad-4fc6-85c8-e4e3043680d2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:47.503784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5943b12c-6dad-4fc6-85c8-e4e3043680d2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:47.505239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5943b12c-6dad-4fc6-85c8-e4e3043680d2 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:47.520693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-5943b12c-6dad-4fc6-85c8-e4e3043680d2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:47.520693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:47.520693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:47.520693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:47.520693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:47.520693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:47.520693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:47.520693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:47.520693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:47.520693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:47.520693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:47.520693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:47.520693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:47.520693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:47.520693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:47.520693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:47.521856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:47.521856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:47.521856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:47.521856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:47.521856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:47.521856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:47.521856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:47.521856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:47.521856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:47.521856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:47.521856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:47.521856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:47.521856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:47.521856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:47.521856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:47.521856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:47.521856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:47.522955 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:47.522955 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:47.522955 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:47.522955 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 544/1089] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:47.527517 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3547ff-aa0f-4986-b120-47e573c01888 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:47.575401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:47.626219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3547ff-aa0f-4986-b120-47e573c01888 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:47.659540 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889fac25-c66f-41a5-affd-556bc0fea36d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:47.660142 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889fac25-c66f-41a5-affd-556bc0fea36d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:47.660426 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889fac25-c66f-41a5-affd-556bc0fea36d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:47.660703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889fac25-c66f-41a5-affd-556bc0fea36d None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:47.661341 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889fac25-c66f-41a5-affd-556bc0fea36d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:47.661644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889fac25-c66f-41a5-affd-556bc0fea36d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:47.663031 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889fac25-c66f-41a5-affd-556bc0fea36d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:47.674494 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 546/1090] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:47.680126 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5943b12c-6dad-4fc6-85c8-e4e3043680d2 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:47.777821 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5943b12c-6dad-4fc6-85c8-e4e3043680d2 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:47.806194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:47.811039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3cdc01-0b0b-4605-b364-9aa4405e93b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:47.811670 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3cdc01-0b0b-4605-b364-9aa4405e93b1 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:47.812501 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3cdc01-0b0b-4605-b364-9aa4405e93b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:47.812859 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3cdc01-0b0b-4605-b364-9aa4405e93b1 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:47.820047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3cdc01-0b0b-4605-b364-9aa4405e93b1 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:47.821485 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3cdc01-0b0b-4605-b364-9aa4405e93b1 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:47.832863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-5d3cdc01-0b0b-4605-b364-9aa4405e93b1 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 16 03:28:47.832863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:47.832863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:47.832863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:47.832863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:47.832863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:47.832863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:47.832863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:47.832863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:47.832863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:47.832863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:47.832863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:47.832863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:28:47.832863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:28:47.832863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:28:47.832863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:28:47.834010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:47.834010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:47.834010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:28:47.834010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:47.834010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:28:47.834010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:28:47.834010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:47.834010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:47.834010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:47.834010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:47.834010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:47.834010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:47.834010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:47.834010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:47.834010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:47.834010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:47.834010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:47.834010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:47.835503 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:47.835503 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:47.835503 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:28:47.835503 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:28:47.835503 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:28:47.835503 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:28:47.835503 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:28:47.835503 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:28:47.835503 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 16 03:28:47.835503 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:47.835503 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 545/1091] 10.222.0.49 () {56 vars in 1093 bytes} [Fri Sep 16 03:28:47 2022] GET /identity/v3/users/nova => generated 82 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:28:47.839643 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889fac25-c66f-41a5-affd-556bc0fea36d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:47.935882 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889fac25-c66f-41a5-affd-556bc0fea36d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:47.945293 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:47.969165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0194256d-0cc0-429f-91d5-5eabdbad0366 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:47.969764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0194256d-0cc0-429f-91d5-5eabdbad0366 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:47.970042 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0194256d-0cc0-429f-91d5-5eabdbad0366 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:47.970314 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0194256d-0cc0-429f-91d5-5eabdbad0366 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:47.971066 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0194256d-0cc0-429f-91d5-5eabdbad0366 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:47.971373 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0194256d-0cc0-429f-91d5-5eabdbad0366 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:47.972779 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0194256d-0cc0-429f-91d5-5eabdbad0366 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:47.989384 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 547/1092] 10.222.0.49 () {56 vars in 1133 bytes} [Fri Sep 16 03:28:47 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:47.995204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3cdc01-0b0b-4605-b364-9aa4405e93b1 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:48.094031 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3cdc01-0b0b-4605-b364-9aa4405e93b1 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:48.126987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66d0c99-24b5-475d-8258-72f6c56c92ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:48.127585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66d0c99-24b5-475d-8258-72f6c56c92ef None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:48.127861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66d0c99-24b5-475d-8258-72f6c56c92ef None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:48.128131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66d0c99-24b5-475d-8258-72f6c56c92ef None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:48.184123 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:48.194311 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66d0c99-24b5-475d-8258-72f6c56c92ef None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:48.197605 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66d0c99-24b5-475d-8258-72f6c56c92ef None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:48.292490 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 546/1093] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:28:47 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 03:28:48.298775 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0194256d-0cc0-429f-91d5-5eabdbad0366 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:48.401414 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0194256d-0cc0-429f-91d5-5eabdbad0366 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:48.434605 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ce75d8-2202-4ac2-b6ee-17c94bde32aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:48.435279 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ce75d8-2202-4ac2-b6ee-17c94bde32aa None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:48.435559 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ce75d8-2202-4ac2-b6ee-17c94bde32aa None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:48.435835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ce75d8-2202-4ac2-b6ee-17c94bde32aa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:48.446411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ce75d8-2202-4ac2-b6ee-17c94bde32aa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:48.447862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ce75d8-2202-4ac2-b6ee-17c94bde32aa None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:48.462767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-c4ce75d8-2202-4ac2-b6ee-17c94bde32aa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:48.462767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:48.462767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:48.462767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:48.462767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:48.462767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:48.462767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:48.462767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:48.462767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:48.462767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:48.462767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:48.462767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:48.462767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:28:48.462767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:28:48.462767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:28:48.462767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:28:48.463835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:48.463835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:48.463835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:48.463835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:48.463835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:48.463835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:48.463835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:48.463835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:48.463835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:48.463835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:48.463835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:48.463835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:48.463835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:48.463835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:48.463835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:48.463835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:48.463835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:28:48.465126 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:28:48.465126 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:28:48.465126 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:48.465126 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 548/1094] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:48.469688 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66d0c99-24b5-475d-8258-72f6c56c92ef None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:48.569497 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:48.578627 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66d0c99-24b5-475d-8258-72f6c56c92ef None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:48.612360 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbb3072-3c9d-4934-b3a0-e819fbdc5f97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:48.612966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbb3072-3c9d-4934-b3a0-e819fbdc5f97 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:48.613250 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbb3072-3c9d-4934-b3a0-e819fbdc5f97 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:48.613524 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbb3072-3c9d-4934-b3a0-e819fbdc5f97 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:48.614169 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbb3072-3c9d-4934-b3a0-e819fbdc5f97 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:48.614476 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbb3072-3c9d-4934-b3a0-e819fbdc5f97 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:48.615880 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbb3072-3c9d-4934-b3a0-e819fbdc5f97 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:48.627706 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 547/1095] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:28:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:48.633254 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ce75d8-2202-4ac2-b6ee-17c94bde32aa None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:48.645689 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06d3beb-ee2c-49e6-a490-87e51755f381 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:48.645971 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06d3beb-ee2c-49e6-a490-87e51755f381 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:48.646255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06d3beb-ee2c-49e6-a490-87e51755f381 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:48.647640 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 548/1096] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:28:48.652643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f579be-19c6-4c33-969d-299f57201766 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:48.652888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f579be-19c6-4c33-969d-299f57201766 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:48.653164 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f579be-19c6-4c33-969d-299f57201766 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:48.654433 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:48.671279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-15f579be-19c6-4c33-969d-299f57201766 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:48.672582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-15f579be-19c6-4c33-969d-299f57201766 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:48.733809 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ce75d8-2202-4ac2-b6ee-17c94bde32aa None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:48.740088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 549/1097] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 03:28:48.769621 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506b6a65-f1fd-4a06-9042-26fdacea3df0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:48.770245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506b6a65-f1fd-4a06-9042-26fdacea3df0 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:48.770521 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506b6a65-f1fd-4a06-9042-26fdacea3df0 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:48.770791 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506b6a65-f1fd-4a06-9042-26fdacea3df0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:48.777355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506b6a65-f1fd-4a06-9042-26fdacea3df0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:48.778764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506b6a65-f1fd-4a06-9042-26fdacea3df0 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:48.789964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-506b6a65-f1fd-4a06-9042-26fdacea3df0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:48.789964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:48.789964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:48.789964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:48.789964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:48.789964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:48.789964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:48.789964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:48.789964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:48.789964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:48.789964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:48.789964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:48.789964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:28:48.789964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:28:48.789964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:28:48.791087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:28:48.791087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:48.791087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:48.791087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:48.791087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:48.791087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:48.791087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:48.791087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:48.791087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:48.791087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:48.791087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:48.791087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:48.791087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:48.791087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:48.791087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:48.791087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:48.791087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:48.792487 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:28:48.792487 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:28:48.792487 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:28:48.792487 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:28:48.792487 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:28:48.792487 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:28:48.792487 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:28:48.792487 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:28:48.792487 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 549/1098] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:28:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:48.797248 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f579be-19c6-4c33-969d-299f57201766 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:48.876357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:48.893510 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f579be-19c6-4c33-969d-299f57201766 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:48.926964 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e69c4b-53f6-4895-b20c-ca0e76097bef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:48.927504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e69c4b-53f6-4895-b20c-ca0e76097bef None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:48.927786 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e69c4b-53f6-4895-b20c-ca0e76097bef None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:48.928060 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e69c4b-53f6-4895-b20c-ca0e76097bef None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:48.928740 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e69c4b-53f6-4895-b20c-ca0e76097bef None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:48.929045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e69c4b-53f6-4895-b20c-ca0e76097bef None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:48.930469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e69c4b-53f6-4895-b20c-ca0e76097bef None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:48.942532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 550/1099] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:28:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:48.948628 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506b6a65-f1fd-4a06-9042-26fdacea3df0 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:49.019999 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e69c4b-53f6-4895-b20c-ca0e76097bef None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:49.029106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:49.051961 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506b6a65-f1fd-4a06-9042-26fdacea3df0 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:49.085600 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c67770-9cfe-466c-9f76-ffaecd9c8c5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:49.086198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c67770-9cfe-466c-9f76-ffaecd9c8c5b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:49.086473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c67770-9cfe-466c-9f76-ffaecd9c8c5b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:49.086746 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c67770-9cfe-466c-9f76-ffaecd9c8c5b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:49.087558 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c67770-9cfe-466c-9f76-ffaecd9c8c5b None admin] RBAC: Adding query filter params (role.id=4b248d39ed014678bfc26b6f0acd7134, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=ded1bcd10e62499a9aaa0ece5aa55c80) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:49.087863 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c67770-9cfe-466c-9f76-ffaecd9c8c5b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:49.089267 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c67770-9cfe-466c-9f76-ffaecd9c8c5b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:49.094831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 550/1100] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:28:48 2022] GET /identity/v3/role_assignments?user.id=ded1bcd10e62499a9aaa0ece5aa55c80&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=4b248d39ed014678bfc26b6f0acd7134 => generated 608 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:28:49.123200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e69c4b-53f6-4895-b20c-ca0e76097bef None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:49.160362 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b0bcc7-a772-48e1-8113-7761afcbd840 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:49.161015 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b0bcc7-a772-48e1-8113-7761afcbd840 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:49.161295 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b0bcc7-a772-48e1-8113-7761afcbd840 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:49.161565 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b0bcc7-a772-48e1-8113-7761afcbd840 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:49.175209 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b0bcc7-a772-48e1-8113-7761afcbd840 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:28:49.180146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-99b0bcc7-a772-48e1-8113-7761afcbd840 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:28:49.180146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:28:49.180146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:28:49.180146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:28:49.180146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:28:49.180146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:28:49.180146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:28:49.180146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:28:49.180146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:28:49.180146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:28:49.180146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:28:49.180146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:28:49.180146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:28:49.180146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:28:49.180146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:28:49.180146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:28:49.181213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:28:49.181213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:28:49.181213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:28:49.181213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:28:49.181213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:28:49.181213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:28:49.181213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:28:49.181213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:28:49.181213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:28:49.181213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:28:49.181213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:28:49.181213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:28:49.181213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:28:49.181213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:28:49.181213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:28:49.181213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:28:49.182170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:28:49.182170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:28:49.182170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:28:49.182170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:28:49.182170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:28:49.182170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:28:49.182170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:28:49.182170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:28:49.182170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:28:49.182170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:28:49.182170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:28:49.182170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:28:49.182170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 551/1101] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:28:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:28:49.188076 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c67770-9cfe-466c-9f76-ffaecd9c8c5b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:49.290342 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c67770-9cfe-466c-9f76-ffaecd9c8c5b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:49.325384 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013fdab7-fc54-4b0d-9821-317b88d1e84f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:49.325991 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013fdab7-fc54-4b0d-9821-317b88d1e84f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:49.326268 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013fdab7-fc54-4b0d-9821-317b88d1e84f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:49.326541 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013fdab7-fc54-4b0d-9821-317b88d1e84f None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:49.327230 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013fdab7-fc54-4b0d-9821-317b88d1e84f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:49.327692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013fdab7-fc54-4b0d-9821-317b88d1e84f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:49.329063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013fdab7-fc54-4b0d-9821-317b88d1e84f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:49.338117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 551/1102] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:28:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:49.344209 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b0bcc7-a772-48e1-8113-7761afcbd840 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:49.347190 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:49.446499 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b0bcc7-a772-48e1-8113-7761afcbd840 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:49.480569 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cd0d46-e7d6-4099-b2e9-a36dfe06a500 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:49.481199 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cd0d46-e7d6-4099-b2e9-a36dfe06a500 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:49.481482 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cd0d46-e7d6-4099-b2e9-a36dfe06a500 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:49.481764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cd0d46-e7d6-4099-b2e9-a36dfe06a500 None admin] PATH_INFO: `/v3/users/3d5a41fdfbce4585953397195fa25c49` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:49.482963 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90cd0d46-e7d6-4099-b2e9-a36dfe06a500 None admin] RBAC: Authorizing `identity:get_user(user_id=3d5a41fdfbce4585953397195fa25c49)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:49.484924 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90cd0d46-e7d6-4099-b2e9-a36dfe06a500 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:49.486511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 552/1103] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:28:49 2022] GET /identity/v3/users/3d5a41fdfbce4585953397195fa25c49 => generated 285 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:49.492490 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013fdab7-fc54-4b0d-9821-317b88d1e84f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:49.496337 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:49.603520 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013fdab7-fc54-4b0d-9821-317b88d1e84f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:49.638946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c874c2-503f-43d3-861a-ad557b4b2edb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:49.639541 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c874c2-503f-43d3-861a-ad557b4b2edb None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:49.639823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c874c2-503f-43d3-861a-ad557b4b2edb None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:49.640147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c874c2-503f-43d3-861a-ad557b4b2edb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:49.708531 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c874c2-503f-43d3-861a-ad557b4b2edb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:49.709897 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c874c2-503f-43d3-861a-ad557b4b2edb None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:49.712586 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:49.811240 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 552/1104] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:28:49 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 03:28:49.817748 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cd0d46-e7d6-4099-b2e9-a36dfe06a500 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:49.887558 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:49.919290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cd0d46-e7d6-4099-b2e9-a36dfe06a500 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:49.955118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cc17d7-46df-425c-8761-5cfc38fc3e99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:49.955754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cc17d7-46df-425c-8761-5cfc38fc3e99 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:49.956031 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cc17d7-46df-425c-8761-5cfc38fc3e99 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:49.956303 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cc17d7-46df-425c-8761-5cfc38fc3e99 None admin] PATH_INFO: `/v3/projects/b95d42ffdb2b4a668f48401c2db87743` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:49.957400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cc17d7-46df-425c-8761-5cfc38fc3e99 None admin] RBAC: Authorizing `identity:get_project(project_id=b95d42ffdb2b4a668f48401c2db87743)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:49.958943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cc17d7-46df-425c-8761-5cfc38fc3e99 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:49.960701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 553/1105] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:28:49 2022] GET /identity/v3/projects/b95d42ffdb2b4a668f48401c2db87743 => generated 311 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:49.966678 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c874c2-503f-43d3-861a-ad557b4b2edb None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:50.063006 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c874c2-503f-43d3-861a-ad557b4b2edb None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:50.065219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:50.095201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46691aba-b5a3-478b-abd9-eb9b41b8066c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:50.095794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46691aba-b5a3-478b-abd9-eb9b41b8066c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:50.096071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46691aba-b5a3-478b-abd9-eb9b41b8066c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:50.096345 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46691aba-b5a3-478b-abd9-eb9b41b8066c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:50.097135 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46691aba-b5a3-478b-abd9-eb9b41b8066c None admin] RBAC: Adding query filter params (role.id=89c4b3a971224d468d33d2dc411230a2, scope.project.id=b95d42ffdb2b4a668f48401c2db87743, user.id=3d5a41fdfbce4585953397195fa25c49) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:50.097397 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46691aba-b5a3-478b-abd9-eb9b41b8066c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:50.098800 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46691aba-b5a3-478b-abd9-eb9b41b8066c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:50.104353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 553/1106] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:28:49 2022] GET /identity/v3/role_assignments?user.id=3d5a41fdfbce4585953397195fa25c49&scope.project.id=b95d42ffdb2b4a668f48401c2db87743&role.id=89c4b3a971224d468d33d2dc411230a2 => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:52.291546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5982ddc7-6db9-4146-a199-977e67f366b6 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:52.292110 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5982ddc7-6db9-4146-a199-977e67f366b6 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:52.292110 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5982ddc7-6db9-4146-a199-977e67f366b6 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:52.293414 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 554/1107] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:52.298590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2918e884-6cd5-4cd6-8eed-d7996565753e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:52.299045 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2918e884-6cd5-4cd6-8eed-d7996565753e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:52.299138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2918e884-6cd5-4cd6-8eed-d7996565753e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:52.317534 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2918e884-6cd5-4cd6-8eed-d7996565753e None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:52.318711 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2918e884-6cd5-4cd6-8eed-d7996565753e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:52.382854 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 554/1108] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:52.661087 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5982ddc7-6db9-4146-a199-977e67f366b6 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:52.760121 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5982ddc7-6db9-4146-a199-977e67f366b6 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:52.791803 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd09b2b-15f6-446d-b46c-12510dfb46f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:52.792410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd09b2b-15f6-446d-b46c-12510dfb46f9 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:52.792687 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd09b2b-15f6-446d-b46c-12510dfb46f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:52.792957 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd09b2b-15f6-446d-b46c-12510dfb46f9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:52.794041 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd09b2b-15f6-446d-b46c-12510dfb46f9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:52.795494 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd09b2b-15f6-446d-b46c-12510dfb46f9 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:52.797586 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 555/1109] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:28:52 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:52.803330 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2918e884-6cd5-4cd6-8eed-d7996565753e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:52.895829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:52.899709 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2918e884-6cd5-4cd6-8eed-d7996565753e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:52.931998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1992c6d-e552-42cd-b39e-02e67fe3eb36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:52.932542 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1992c6d-e552-42cd-b39e-02e67fe3eb36 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:52.932815 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1992c6d-e552-42cd-b39e-02e67fe3eb36 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:52.933082 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1992c6d-e552-42cd-b39e-02e67fe3eb36 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:52.933715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1992c6d-e552-42cd-b39e-02e67fe3eb36 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:52.935235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1992c6d-e552-42cd-b39e-02e67fe3eb36 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:53.059316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 555/1110] 10.222.0.49 () {60 vars in 1132 bytes} [Fri Sep 16 03:28:52 2022] POST /identity/v3/users => generated 286 bytes in 256 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:28:55.251464 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fef29f2-178e-43b6-9323-d93f376945af None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:55.252038 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fef29f2-178e-43b6-9323-d93f376945af None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:55.252038 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fef29f2-178e-43b6-9323-d93f376945af None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:55.253336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 556/1111] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:55.258702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647af5ed-895f-4865-9c21-9e0bce77bb31 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:55.259151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647af5ed-895f-4865-9c21-9e0bce77bb31 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:55.259291 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647af5ed-895f-4865-9c21-9e0bce77bb31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:55.277546 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-647af5ed-895f-4865-9c21-9e0bce77bb31 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:55.278742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-647af5ed-895f-4865-9c21-9e0bce77bb31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:55.344436 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 556/1112] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:55.619530 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fef29f2-178e-43b6-9323-d93f376945af None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:55.628851 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:55.717367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fef29f2-178e-43b6-9323-d93f376945af None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:55.749742 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6dc6b3-b61c-4d3a-b201-8c810436e6f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:55.750358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6dc6b3-b61c-4d3a-b201-8c810436e6f0 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:55.750640 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6dc6b3-b61c-4d3a-b201-8c810436e6f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:55.750933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6dc6b3-b61c-4d3a-b201-8c810436e6f0 None admin] PATH_INFO: `/v3/roles/fa8bdf568182483aac8a14ef98a12a95` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:55.752301 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6dc6b3-b61c-4d3a-b201-8c810436e6f0 None admin] RBAC: Authorizing `identity:get_role(role_id=fa8bdf568182483aac8a14ef98a12a95)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:55.753657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6dc6b3-b61c-4d3a-b201-8c810436e6f0 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:55.754981 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 557/1113] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:28:55 2022] GET /identity/v3/roles/fa8bdf568182483aac8a14ef98a12a95 => generated 223 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:55.760664 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647af5ed-895f-4865-9c21-9e0bce77bb31 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:55.841368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:55.856600 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647af5ed-895f-4865-9c21-9e0bce77bb31 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:55.888646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd17909-d94c-4dbe-bcb5-d523a4aea2a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:55.889207 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd17909-d94c-4dbe-bcb5-d523a4aea2a5 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:55.889482 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd17909-d94c-4dbe-bcb5-d523a4aea2a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:55.890272 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd17909-d94c-4dbe-bcb5-d523a4aea2a5 None admin] PATH_INFO: `/v3/users/8636d2ab9468485aa6fd109f2df31ee5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:55.903321 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd17909-d94c-4dbe-bcb5-d523a4aea2a5 None admin] RBAC: Authorizing `identity:get_user(user_id=8636d2ab9468485aa6fd109f2df31ee5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:55.904734 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd17909-d94c-4dbe-bcb5-d523a4aea2a5 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:55.907108 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 557/1114] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:28:55 2022] GET /identity/v3/users/8636d2ab9468485aa6fd109f2df31ee5 => generated 286 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:55.912816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6dc6b3-b61c-4d3a-b201-8c810436e6f0 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:55.984878 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:56.008777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6dc6b3-b61c-4d3a-b201-8c810436e6f0 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:56.042016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d268d3-4cf5-4bdc-92e3-8679f61b95c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:56.042617 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d268d3-4cf5-4bdc-92e3-8679f61b95c0 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:56.042898 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d268d3-4cf5-4bdc-92e3-8679f61b95c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:56.043204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d268d3-4cf5-4bdc-92e3-8679f61b95c0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:56.107260 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d268d3-4cf5-4bdc-92e3-8679f61b95c0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:56.108756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d268d3-4cf5-4bdc-92e3-8679f61b95c0 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:56.179483 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:56.204090 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 558/1115] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:28:55 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 03:28:56.210188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd17909-d94c-4dbe-bcb5-d523a4aea2a5 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:56.307584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd17909-d94c-4dbe-bcb5-d523a4aea2a5 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:56.341348 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28123520-516f-4d0c-af4e-6467403f48bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:56.341981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28123520-516f-4d0c-af4e-6467403f48bf None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:56.342265 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28123520-516f-4d0c-af4e-6467403f48bf None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:56.342549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28123520-516f-4d0c-af4e-6467403f48bf None admin] PATH_INFO: `/v3/projects/b95d42ffdb2b4a668f48401c2db87743` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:56.343799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28123520-516f-4d0c-af4e-6467403f48bf None admin] RBAC: Authorizing `identity:get_project(project_id=b95d42ffdb2b4a668f48401c2db87743)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:56.345221 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28123520-516f-4d0c-af4e-6467403f48bf None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:56.346833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 558/1116] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:28:56 2022] GET /identity/v3/projects/b95d42ffdb2b4a668f48401c2db87743 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:56.352885 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d268d3-4cf5-4bdc-92e3-8679f61b95c0 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:56.452367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d268d3-4cf5-4bdc-92e3-8679f61b95c0 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:56.485593 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6761e2-600a-44c8-9ea3-398c221a5247 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:56.486195 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6761e2-600a-44c8-9ea3-398c221a5247 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:56.486476 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6761e2-600a-44c8-9ea3-398c221a5247 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:56.486752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6761e2-600a-44c8-9ea3-398c221a5247 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:56.487508 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6761e2-600a-44c8-9ea3-398c221a5247 None admin] RBAC: Adding query filter params (role.id=fa8bdf568182483aac8a14ef98a12a95, scope.project.id=b95d42ffdb2b4a668f48401c2db87743, user.id=8636d2ab9468485aa6fd109f2df31ee5) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:28:56.487819 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6761e2-600a-44c8-9ea3-398c221a5247 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:56.489243 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6761e2-600a-44c8-9ea3-398c221a5247 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:56.494396 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 559/1117] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:28:56 2022] GET /identity/v3/role_assignments?user.id=8636d2ab9468485aa6fd109f2df31ee5&scope.project.id=b95d42ffdb2b4a668f48401c2db87743&role.id=fa8bdf568182483aac8a14ef98a12a95 => generated 259 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:58.609048 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4403469f-b379-4616-8a44-0e7d1b8276b1 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:58.609694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4403469f-b379-4616-8a44-0e7d1b8276b1 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:58.609694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4403469f-b379-4616-8a44-0e7d1b8276b1 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:58.611034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 559/1118] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:28:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:28:58.616180 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47a7cee-c05f-4475-aacb-519b9ea62041 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:58.616566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47a7cee-c05f-4475-aacb-519b9ea62041 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:58.616729 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47a7cee-c05f-4475-aacb-519b9ea62041 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:58.634981 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-b47a7cee-c05f-4475-aacb-519b9ea62041 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:28:58.636151 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b47a7cee-c05f-4475-aacb-519b9ea62041 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:28:58.668684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:58.704952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 560/1119] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:28:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:28:58.987164 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4403469f-b379-4616-8a44-0e7d1b8276b1 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:58.999284 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:59.085146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4403469f-b379-4616-8a44-0e7d1b8276b1 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:59.118179 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e59f35-4716-4b29-991e-4b31401ccc86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:59.118794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e59f35-4716-4b29-991e-4b31401ccc86 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:59.119106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e59f35-4716-4b29-991e-4b31401ccc86 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:59.119384 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e59f35-4716-4b29-991e-4b31401ccc86 None admin] PATH_INFO: `/v3/roles/fa8bdf568182483aac8a14ef98a12a95` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:59.120756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e59f35-4716-4b29-991e-4b31401ccc86 None admin] RBAC: Authorizing `identity:get_role(role_id=fa8bdf568182483aac8a14ef98a12a95)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:59.122108 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e59f35-4716-4b29-991e-4b31401ccc86 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:59.123422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 560/1120] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:28:58 2022] GET /identity/v3/roles/fa8bdf568182483aac8a14ef98a12a95 => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:59.129434 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47a7cee-c05f-4475-aacb-519b9ea62041 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:59.224928 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47a7cee-c05f-4475-aacb-519b9ea62041 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:59.257936 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294b0d00-0659-42e3-9c45-7eb85818825c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:59.258496 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294b0d00-0659-42e3-9c45-7eb85818825c None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:59.258777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294b0d00-0659-42e3-9c45-7eb85818825c None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:59.259080 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294b0d00-0659-42e3-9c45-7eb85818825c None admin] PATH_INFO: `/v3/users/8636d2ab9468485aa6fd109f2df31ee5` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:59.260174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294b0d00-0659-42e3-9c45-7eb85818825c None admin] RBAC: Authorizing `identity:get_user(user_id=8636d2ab9468485aa6fd109f2df31ee5)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:59.261594 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294b0d00-0659-42e3-9c45-7eb85818825c None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:59.263156 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 561/1121] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:28:59 2022] GET /identity/v3/users/8636d2ab9468485aa6fd109f2df31ee5 => generated 286 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:59.268821 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e59f35-4716-4b29-991e-4b31401ccc86 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:59.349505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:59.370341 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e59f35-4716-4b29-991e-4b31401ccc86 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:59.404455 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e470edc-46db-40ae-b5b4-2bf2380bd7cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:59.405099 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e470edc-46db-40ae-b5b4-2bf2380bd7cb None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:59.405399 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e470edc-46db-40ae-b5b4-2bf2380bd7cb None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:59.406504 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e470edc-46db-40ae-b5b4-2bf2380bd7cb None admin] PATH_INFO: `/v3/projects/b95d42ffdb2b4a668f48401c2db87743` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:59.407752 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e470edc-46db-40ae-b5b4-2bf2380bd7cb None admin] RBAC: Authorizing `identity:get_project(project_id=b95d42ffdb2b4a668f48401c2db87743)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:59.409177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e470edc-46db-40ae-b5b4-2bf2380bd7cb None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:59.410818 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 561/1122] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:28:59 2022] GET /identity/v3/projects/b95d42ffdb2b4a668f48401c2db87743 => generated 311 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:28:59.416933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294b0d00-0659-42e3-9c45-7eb85818825c None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:28:59.451819 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:28:59.517836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294b0d00-0659-42e3-9c45-7eb85818825c None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:28:59.552440 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3ee4cf-a5b8-40c7-ab24-b98f95cfdf3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:28:59.553094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3ee4cf-a5b8-40c7-ab24-b98f95cfdf3b None admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:28:59.553370 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3ee4cf-a5b8-40c7-ab24-b98f95cfdf3b None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:28:59.553642 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3ee4cf-a5b8-40c7-ab24-b98f95cfdf3b None admin] PATH_INFO: `/v3/projects/b95d42ffdb2b4a668f48401c2db87743/users/8636d2ab9468485aa6fd109f2df31ee5/roles/fa8bdf568182483aac8a14ef98a12a95` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:28:59.555747 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3ee4cf-a5b8-40c7-ab24-b98f95cfdf3b None admin] RBAC: Authorizing `identity:create_grant(project_id=b95d42ffdb2b4a668f48401c2db87743, user_id=8636d2ab9468485aa6fd109f2df31ee5, role_id=fa8bdf568182483aac8a14ef98a12a95)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:28:59.557175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3ee4cf-a5b8-40c7-ab24-b98f95cfdf3b None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:28:59.668174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 562/1123] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:28:59 2022] PUT /identity/v3/projects/b95d42ffdb2b4a668f48401c2db87743/users/8636d2ab9468485aa6fd109f2df31ee5/roles/fa8bdf568182483aac8a14ef98a12a95 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:29:01.775455 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c406c28d-ce66-42ff-bd22-6b35262fb57d None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:01.775985 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c406c28d-ce66-42ff-bd22-6b35262fb57d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:01.775985 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c406c28d-ce66-42ff-bd22-6b35262fb57d None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:01.777319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 562/1124] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:29:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:29:01.783068 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1633fe9b-49df-45cc-a91c-b0498aed0760 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:01.783469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1633fe9b-49df-45cc-a91c-b0498aed0760 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:01.783626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1633fe9b-49df-45cc-a91c-b0498aed0760 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:01.801843 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-1633fe9b-49df-45cc-a91c-b0498aed0760 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:29:01.803070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1633fe9b-49df-45cc-a91c-b0498aed0760 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:29:01.875743 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 563/1125] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:29:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:29:02.151446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c406c28d-ce66-42ff-bd22-6b35262fb57d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:02.250452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c406c28d-ce66-42ff-bd22-6b35262fb57d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:02.284005 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7001c4-5d1a-447f-b89e-917d8b4bc642 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:02.284623 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7001c4-5d1a-447f-b89e-917d8b4bc642 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:02.284907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7001c4-5d1a-447f-b89e-917d8b4bc642 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:02.285185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7001c4-5d1a-447f-b89e-917d8b4bc642 None admin] PATH_INFO: `/v3/roles/fa8bdf568182483aac8a14ef98a12a95` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:02.286580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c7001c4-5d1a-447f-b89e-917d8b4bc642 None admin] RBAC: Authorizing `identity:get_role(role_id=fa8bdf568182483aac8a14ef98a12a95)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:02.287964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c7001c4-5d1a-447f-b89e-917d8b4bc642 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:02.289261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 563/1126] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:29:02 2022] GET /identity/v3/roles/fa8bdf568182483aac8a14ef98a12a95 => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:02.295118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1633fe9b-49df-45cc-a91c-b0498aed0760 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:02.321408 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:02.391648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1633fe9b-49df-45cc-a91c-b0498aed0760 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:02.425265 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6f9de5-02a1-452f-a9d1-b6b7704e8d68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:02.425836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6f9de5-02a1-452f-a9d1-b6b7704e8d68 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:02.426119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6f9de5-02a1-452f-a9d1-b6b7704e8d68 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:02.426396 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6f9de5-02a1-452f-a9d1-b6b7704e8d68 None admin] PATH_INFO: `/v3/users/8636d2ab9468485aa6fd109f2df31ee5` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:02.427509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6f9de5-02a1-452f-a9d1-b6b7704e8d68 None admin] RBAC: Authorizing `identity:get_user(user_id=8636d2ab9468485aa6fd109f2df31ee5)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:02.428931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6f9de5-02a1-452f-a9d1-b6b7704e8d68 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:02.430493 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 564/1127] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:29:02 2022] GET /identity/v3/users/8636d2ab9468485aa6fd109f2df31ee5 => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:02.436268 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7001c4-5d1a-447f-b89e-917d8b4bc642 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:02.451551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:02.541069 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7001c4-5d1a-447f-b89e-917d8b4bc642 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:02.574151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f05cffd-f99b-443d-a095-06ae7963ff4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:02.574748 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f05cffd-f99b-443d-a095-06ae7963ff4c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:02.575062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f05cffd-f99b-443d-a095-06ae7963ff4c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:02.575341 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f05cffd-f99b-443d-a095-06ae7963ff4c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:02.639483 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f05cffd-f99b-443d-a095-06ae7963ff4c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:02.641044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f05cffd-f99b-443d-a095-06ae7963ff4c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:02.654968 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:02.743743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 564/1128] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:29:02 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 03:29:02.750043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6f9de5-02a1-452f-a9d1-b6b7704e8d68 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:02.840505 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:02.847564 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6f9de5-02a1-452f-a9d1-b6b7704e8d68 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:02.880842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907a7d9a-e296-41f1-aaf2-8ec15feb0e36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:02.881477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907a7d9a-e296-41f1-aaf2-8ec15feb0e36 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:02.881746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907a7d9a-e296-41f1-aaf2-8ec15feb0e36 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:02.882023 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907a7d9a-e296-41f1-aaf2-8ec15feb0e36 None admin] PATH_INFO: `/v3/projects/b95d42ffdb2b4a668f48401c2db87743` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:02.883171 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907a7d9a-e296-41f1-aaf2-8ec15feb0e36 None admin] RBAC: Authorizing `identity:get_project(project_id=b95d42ffdb2b4a668f48401c2db87743)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:02.884596 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907a7d9a-e296-41f1-aaf2-8ec15feb0e36 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:02.886192 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 565/1129] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:29:02 2022] GET /identity/v3/projects/b95d42ffdb2b4a668f48401c2db87743 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:02.895262 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f05cffd-f99b-443d-a095-06ae7963ff4c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:03.004262 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f05cffd-f99b-443d-a095-06ae7963ff4c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:03.017327 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:03.036700 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4a58f8-4eb8-45ac-bbc9-4b0d92bd6d96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:03.037301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4a58f8-4eb8-45ac-bbc9-4b0d92bd6d96 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:03.037566 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4a58f8-4eb8-45ac-bbc9-4b0d92bd6d96 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:03.037836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4a58f8-4eb8-45ac-bbc9-4b0d92bd6d96 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:03.038544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4a58f8-4eb8-45ac-bbc9-4b0d92bd6d96 None admin] RBAC: Adding query filter params (role.id=fa8bdf568182483aac8a14ef98a12a95, scope.project.id=b95d42ffdb2b4a668f48401c2db87743, user.id=8636d2ab9468485aa6fd109f2df31ee5) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:29:03.038848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4a58f8-4eb8-45ac-bbc9-4b0d92bd6d96 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:03.040287 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4a58f8-4eb8-45ac-bbc9-4b0d92bd6d96 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:03.046993 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 565/1130] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:29:02 2022] GET /identity/v3/role_assignments?user.id=8636d2ab9468485aa6fd109f2df31ee5&scope.project.id=b95d42ffdb2b4a668f48401c2db87743&role.id=fa8bdf568182483aac8a14ef98a12a95 => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:05.191859 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1f1596-68d1-484a-a626-519c9cbff11a None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:05.192427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1f1596-68d1-484a-a626-519c9cbff11a None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:05.192427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1f1596-68d1-484a-a626-519c9cbff11a None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:05.193725 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 566/1131] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:29:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:29:05.199124 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361209d3-1335-4c62-b8ae-338eb492d409 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:05.199503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361209d3-1335-4c62-b8ae-338eb492d409 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:05.199673 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361209d3-1335-4c62-b8ae-338eb492d409 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:05.217624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-361209d3-1335-4c62-b8ae-338eb492d409 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:29:05.219385 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-361209d3-1335-4c62-b8ae-338eb492d409 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:29:05.282592 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 566/1132] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:29:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:29:05.561930 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1f1596-68d1-484a-a626-519c9cbff11a None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:05.663961 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1f1596-68d1-484a-a626-519c9cbff11a None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:05.697821 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8cd4cc-1c08-4d39-9233-1780aeeac02d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:05.698445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8cd4cc-1c08-4d39-9233-1780aeeac02d None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:05.698722 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8cd4cc-1c08-4d39-9233-1780aeeac02d None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:05.699024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8cd4cc-1c08-4d39-9233-1780aeeac02d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:05.700119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8cd4cc-1c08-4d39-9233-1780aeeac02d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:05.701542 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8cd4cc-1c08-4d39-9233-1780aeeac02d None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:05.703152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 567/1133] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:29:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:05.709216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361209d3-1335-4c62-b8ae-338eb492d409 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:05.804593 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361209d3-1335-4c62-b8ae-338eb492d409 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:05.813567 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:05.836575 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5666ee3f-4740-47da-b26d-68257f414166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:05.837170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5666ee3f-4740-47da-b26d-68257f414166 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:05.837445 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5666ee3f-4740-47da-b26d-68257f414166 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:05.837716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5666ee3f-4740-47da-b26d-68257f414166 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:05.838345 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5666ee3f-4740-47da-b26d-68257f414166 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:05.839789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5666ee3f-4740-47da-b26d-68257f414166 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:05.919617 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 567/1134] 10.222.0.49 () {60 vars in 1141 bytes} [Fri Sep 16 03:29:05 2022] POST /identity/v3/projects => generated 311 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:29:08.115079 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e03ca9-3f9c-40c4-86df-b66c20859e6c None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:08.115633 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e03ca9-3f9c-40c4-86df-b66c20859e6c None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:08.115633 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e03ca9-3f9c-40c4-86df-b66c20859e6c None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:08.116941 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 568/1135] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:29:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:29:08.122072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb8b796-1f1c-47ca-bf05-d79cfcecea3f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:08.122497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb8b796-1f1c-47ca-bf05-d79cfcecea3f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:08.122630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb8b796-1f1c-47ca-bf05-d79cfcecea3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:08.140536 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7eb8b796-1f1c-47ca-bf05-d79cfcecea3f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:29:08.141708 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7eb8b796-1f1c-47ca-bf05-d79cfcecea3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:29:08.205496 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 568/1136] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:29:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 03:29:08.476788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e03ca9-3f9c-40c4-86df-b66c20859e6c None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:08.507321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:08.576243 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e03ca9-3f9c-40c4-86df-b66c20859e6c None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:08.609224 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255ae65f-e013-4d75-bad7-5a2b48ba7382 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:08.609839 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255ae65f-e013-4d75-bad7-5a2b48ba7382 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:08.610115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255ae65f-e013-4d75-bad7-5a2b48ba7382 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:08.610384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255ae65f-e013-4d75-bad7-5a2b48ba7382 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:08.611494 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255ae65f-e013-4d75-bad7-5a2b48ba7382 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:08.612920 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255ae65f-e013-4d75-bad7-5a2b48ba7382 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:08.614445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 569/1137] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:29:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:08.620193 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb8b796-1f1c-47ca-bf05-d79cfcecea3f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:08.710110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:08.720329 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb8b796-1f1c-47ca-bf05-d79cfcecea3f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:08.753955 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d0742b-2876-47ae-83ec-750962bc3f50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:08.754491 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d0742b-2876-47ae-83ec-750962bc3f50 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:08.754771 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d0742b-2876-47ae-83ec-750962bc3f50 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:08.755087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d0742b-2876-47ae-83ec-750962bc3f50 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:08.755735 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d0742b-2876-47ae-83ec-750962bc3f50 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:08.757161 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d0742b-2876-47ae-83ec-750962bc3f50 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:08.863891 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 569/1138] 10.222.0.49 () {60 vars in 1132 bytes} [Fri Sep 16 03:29:08 2022] POST /identity/v3/users => generated 286 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:29:11.063897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafaaa99-9ca1-4d4d-9e32-659d11374969 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:11.064469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafaaa99-9ca1-4d4d-9e32-659d11374969 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:11.064469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafaaa99-9ca1-4d4d-9e32-659d11374969 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:11.065770 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 570/1139] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:29:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:29:11.070947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a206524-2da4-4d84-abf7-e5a26d4966bf None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:11.071392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a206524-2da4-4d84-abf7-e5a26d4966bf None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:11.071494 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a206524-2da4-4d84-abf7-e5a26d4966bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:11.089702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-3a206524-2da4-4d84-abf7-e5a26d4966bf None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:29:11.090864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3a206524-2da4-4d84-abf7-e5a26d4966bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:29:11.154789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 570/1140] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:29:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:29:11.424925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafaaa99-9ca1-4d4d-9e32-659d11374969 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:11.518828 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:11.523400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafaaa99-9ca1-4d4d-9e32-659d11374969 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:11.556024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1931445c-1ab6-4a68-a385-7b71d3e25695 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:11.556637 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1931445c-1ab6-4a68-a385-7b71d3e25695 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:11.556915 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1931445c-1ab6-4a68-a385-7b71d3e25695 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:11.557184 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1931445c-1ab6-4a68-a385-7b71d3e25695 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:11.566946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1931445c-1ab6-4a68-a385-7b71d3e25695 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:29:11.571711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-1931445c-1ab6-4a68-a385-7b71d3e25695 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:29:11.571711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:29:11.571711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:29:11.571711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:29:11.571711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:29:11.571711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:29:11.571711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:29:11.571711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:29:11.571711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:29:11.571711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:29:11.571711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:29:11.571711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:29:11.571711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:29:11.571711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:29:11.571711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:29:11.571711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:29:11.572888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:29:11.572888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:29:11.572888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:29:11.572888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:29:11.572888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:29:11.572888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:29:11.572888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:29:11.572888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:29:11.572888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:29:11.572888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:29:11.572888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:29:11.572888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:29:11.572888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:29:11.572888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:29:11.572888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:29:11.572888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:29:11.574030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:29:11.574030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:29:11.574030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:29:11.574030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:29:11.574030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:29:11.574030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:29:11.574030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:29:11.574030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:29:11.574030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:29:11.574030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:29:11.574030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:29:11.574030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:29:11.575037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 571/1141] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:29:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:29:11.579564 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a206524-2da4-4d84-abf7-e5a26d4966bf None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:11.650311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:11.672236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a206524-2da4-4d84-abf7-e5a26d4966bf None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:11.705049 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3730b48-39ab-483d-86dd-67461b615dca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:11.705643 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3730b48-39ab-483d-86dd-67461b615dca None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:11.705926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3730b48-39ab-483d-86dd-67461b615dca None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:11.706192 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3730b48-39ab-483d-86dd-67461b615dca None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:11.706843 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3730b48-39ab-483d-86dd-67461b615dca None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:29:11.707183 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3730b48-39ab-483d-86dd-67461b615dca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:11.708512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3730b48-39ab-483d-86dd-67461b615dca None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:11.716572 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 571/1142] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:29:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:11.722800 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1931445c-1ab6-4a68-a385-7b71d3e25695 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:11.823718 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1931445c-1ab6-4a68-a385-7b71d3e25695 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:11.856577 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b70c6be-0a1e-4b4a-a85b-b537727a4898 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:11.857134 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b70c6be-0a1e-4b4a-a85b-b537727a4898 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:11.857411 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b70c6be-0a1e-4b4a-a85b-b537727a4898 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:11.857681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b70c6be-0a1e-4b4a-a85b-b537727a4898 None admin] PATH_INFO: `/v3/users/e0b46d861a2e431da439d1a890c98306` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:11.870065 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b70c6be-0a1e-4b4a-a85b-b537727a4898 None admin] RBAC: Authorizing `identity:get_user(user_id=e0b46d861a2e431da439d1a890c98306)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:11.871520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b70c6be-0a1e-4b4a-a85b-b537727a4898 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:11.873083 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 572/1143] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:29:11 2022] GET /identity/v3/users/e0b46d861a2e431da439d1a890c98306 => generated 286 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:11.879749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3730b48-39ab-483d-86dd-67461b615dca None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:11.976198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3730b48-39ab-483d-86dd-67461b615dca None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:12.008556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f9794f-b80d-459c-94f5-4e8af82a32f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:12.009145 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f9794f-b80d-459c-94f5-4e8af82a32f8 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:12.009426 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f9794f-b80d-459c-94f5-4e8af82a32f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:12.009700 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f9794f-b80d-459c-94f5-4e8af82a32f8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:12.018377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:12.074672 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f9794f-b80d-459c-94f5-4e8af82a32f8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:12.076159 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f9794f-b80d-459c-94f5-4e8af82a32f8 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:12.168011 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 572/1144] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:29:11 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 03:29:12.174433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b70c6be-0a1e-4b4a-a85b-b537727a4898 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:12.192304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:12.269772 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b70c6be-0a1e-4b4a-a85b-b537727a4898 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:12.302349 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757b13e3-6028-4ddd-8c9c-948e160bb43c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:12.302986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757b13e3-6028-4ddd-8c9c-948e160bb43c None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:12.303267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757b13e3-6028-4ddd-8c9c-948e160bb43c None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:12.303539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757b13e3-6028-4ddd-8c9c-948e160bb43c None admin] PATH_INFO: `/v3/projects/18dfa1ffdfd649cdb3f6c3a41a27c312` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:12.304626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757b13e3-6028-4ddd-8c9c-948e160bb43c None admin] RBAC: Authorizing `identity:get_project(project_id=18dfa1ffdfd649cdb3f6c3a41a27c312)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:12.306038 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757b13e3-6028-4ddd-8c9c-948e160bb43c None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:12.307699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 573/1145] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:29:12 2022] GET /identity/v3/projects/18dfa1ffdfd649cdb3f6c3a41a27c312 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:12.314533 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f9794f-b80d-459c-94f5-4e8af82a32f8 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:12.351502 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:12.408591 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f9794f-b80d-459c-94f5-4e8af82a32f8 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:12.441519 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aebb4f3-b715-4837-a84a-b002d09ee559 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:12.442052 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aebb4f3-b715-4837-a84a-b002d09ee559 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:12.442327 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aebb4f3-b715-4837-a84a-b002d09ee559 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:12.442597 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aebb4f3-b715-4837-a84a-b002d09ee559 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:12.443337 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aebb4f3-b715-4837-a84a-b002d09ee559 None admin] RBAC: Adding query filter params (role.id=89c4b3a971224d468d33d2dc411230a2, scope.project.id=18dfa1ffdfd649cdb3f6c3a41a27c312, user.id=e0b46d861a2e431da439d1a890c98306) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:29:12.443641 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aebb4f3-b715-4837-a84a-b002d09ee559 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:12.445040 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aebb4f3-b715-4837-a84a-b002d09ee559 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:12.450290 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 573/1146] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:29:12 2022] GET /identity/v3/role_assignments?user.id=e0b46d861a2e431da439d1a890c98306&scope.project.id=18dfa1ffdfd649cdb3f6c3a41a27c312&role.id=89c4b3a971224d468d33d2dc411230a2 => generated 259 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:14.550643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005caec9-9ae6-454a-a915-fa6ec896da99 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:14.551220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005caec9-9ae6-454a-a915-fa6ec896da99 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:14.551220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005caec9-9ae6-454a-a915-fa6ec896da99 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:14.552564 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 574/1147] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:29:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:29:14.557686 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72af5dad-ad28-410f-a7fa-d3be4a1c19ca None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:14.558124 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72af5dad-ad28-410f-a7fa-d3be4a1c19ca None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:14.558232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72af5dad-ad28-410f-a7fa-d3be4a1c19ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:14.576647 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-72af5dad-ad28-410f-a7fa-d3be4a1c19ca None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:29:14.577889 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-72af5dad-ad28-410f-a7fa-d3be4a1c19ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:29:14.643104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 574/1148] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:29:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:29:14.928324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005caec9-9ae6-454a-a915-fa6ec896da99 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:15.047835 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:15.068965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005caec9-9ae6-454a-a915-fa6ec896da99 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:15.112514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42dd6e5e-3604-42da-8af8-44a0034ff6fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:15.114199 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dd6e5e-3604-42da-8af8-44a0034ff6fb None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:15.114492 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dd6e5e-3604-42da-8af8-44a0034ff6fb None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:15.114774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dd6e5e-3604-42da-8af8-44a0034ff6fb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:15.128128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42dd6e5e-3604-42da-8af8-44a0034ff6fb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:29:15.133433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-42dd6e5e-3604-42da-8af8-44a0034ff6fb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:29:15.133433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:29:15.133433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:29:15.133433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:29:15.133433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:29:15.133433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:29:15.133433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:29:15.133433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:29:15.133433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:29:15.133433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:29:15.133433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:29:15.133433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:29:15.133433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:29:15.133433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:29:15.133433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:29:15.133433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:29:15.134534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:29:15.134534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:29:15.134534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:29:15.134534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:29:15.134534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:29:15.134534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:29:15.134534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:29:15.134534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:29:15.134534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:29:15.134534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:29:15.134534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:29:15.134534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:29:15.134534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:29:15.134534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:29:15.134534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:29:15.134534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:29:15.135685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:29:15.135685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:29:15.135685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:29:15.135685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:29:15.135685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:29:15.135685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:29:15.135685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:29:15.135685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:29:15.135685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:29:15.135685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:29:15.135685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:29:15.135685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:29:15.135685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 575/1149] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:29:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:29:15.144837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72af5dad-ad28-410f-a7fa-d3be4a1c19ca None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:15.177919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:15.247525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72af5dad-ad28-410f-a7fa-d3be4a1c19ca None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:15.283153 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f85aa38-bd7e-4915-9a8e-5e8307782c20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:15.283761 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f85aa38-bd7e-4915-9a8e-5e8307782c20 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:15.284035 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f85aa38-bd7e-4915-9a8e-5e8307782c20 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:15.284302 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f85aa38-bd7e-4915-9a8e-5e8307782c20 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:15.284958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f85aa38-bd7e-4915-9a8e-5e8307782c20 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:29:15.285255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f85aa38-bd7e-4915-9a8e-5e8307782c20 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:15.286601 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f85aa38-bd7e-4915-9a8e-5e8307782c20 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:15.295428 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 575/1150] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:29:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:15.301445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42dd6e5e-3604-42da-8af8-44a0034ff6fb None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:15.422958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42dd6e5e-3604-42da-8af8-44a0034ff6fb None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:15.458662 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445e4384-985b-4490-86d5-a8882d3f5587 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:15.459371 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445e4384-985b-4490-86d5-a8882d3f5587 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:15.459653 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445e4384-985b-4490-86d5-a8882d3f5587 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:15.459943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445e4384-985b-4490-86d5-a8882d3f5587 None admin] PATH_INFO: `/v3/users/e0b46d861a2e431da439d1a890c98306` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:15.461070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445e4384-985b-4490-86d5-a8882d3f5587 None admin] RBAC: Authorizing `identity:get_user(user_id=e0b46d861a2e431da439d1a890c98306)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:15.462494 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445e4384-985b-4490-86d5-a8882d3f5587 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:15.464114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 576/1151] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:29:15 2022] GET /identity/v3/users/e0b46d861a2e431da439d1a890c98306 => generated 286 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:15.471451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f85aa38-bd7e-4915-9a8e-5e8307782c20 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:15.566243 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:15.572602 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f85aa38-bd7e-4915-9a8e-5e8307782c20 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:15.607241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c8ff44-9177-4f49-8374-aeb98fa00fba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:15.607884 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c8ff44-9177-4f49-8374-aeb98fa00fba None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:15.608155 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c8ff44-9177-4f49-8374-aeb98fa00fba None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:15.608449 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c8ff44-9177-4f49-8374-aeb98fa00fba None admin] PATH_INFO: `/v3/projects/18dfa1ffdfd649cdb3f6c3a41a27c312` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:15.609563 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c8ff44-9177-4f49-8374-aeb98fa00fba None admin] RBAC: Authorizing `identity:get_project(project_id=18dfa1ffdfd649cdb3f6c3a41a27c312)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:15.611092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c8ff44-9177-4f49-8374-aeb98fa00fba None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:15.612868 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 576/1152] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:29:15 2022] GET /identity/v3/projects/18dfa1ffdfd649cdb3f6c3a41a27c312 => generated 311 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:15.619082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445e4384-985b-4490-86d5-a8882d3f5587 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:15.637109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:15.727668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445e4384-985b-4490-86d5-a8882d3f5587 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:15.763040 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14175916-d673-486f-83ab-9a228e3fdf56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:15.763746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14175916-d673-486f-83ab-9a228e3fdf56 None admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:15.764127 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14175916-d673-486f-83ab-9a228e3fdf56 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:15.764403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14175916-d673-486f-83ab-9a228e3fdf56 None admin] PATH_INFO: `/v3/projects/18dfa1ffdfd649cdb3f6c3a41a27c312/users/e0b46d861a2e431da439d1a890c98306/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:15.767394 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14175916-d673-486f-83ab-9a228e3fdf56 None admin] RBAC: Authorizing `identity:create_grant(project_id=18dfa1ffdfd649cdb3f6c3a41a27c312, user_id=e0b46d861a2e431da439d1a890c98306, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:15.768810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14175916-d673-486f-83ab-9a228e3fdf56 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:15.869915 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 577/1153] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:29:15 2022] PUT /identity/v3/projects/18dfa1ffdfd649cdb3f6c3a41a27c312/users/e0b46d861a2e431da439d1a890c98306/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:29:18.010378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633ee38b-064e-470e-9a55-779e5a10e01a None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:18.010950 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633ee38b-064e-470e-9a55-779e5a10e01a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:18.010950 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633ee38b-064e-470e-9a55-779e5a10e01a None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:18.012294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 577/1154] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:29:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:29:18.017612 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d795c26d-53fd-4430-a8b4-e63f4fd0b694 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:18.018043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d795c26d-53fd-4430-a8b4-e63f4fd0b694 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:18.018165 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d795c26d-53fd-4430-a8b4-e63f4fd0b694 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:18.036855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-d795c26d-53fd-4430-a8b4-e63f4fd0b694 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:29:18.038031 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d795c26d-53fd-4430-a8b4-e63f4fd0b694 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:29:18.102621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 578/1155] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:29:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:29:18.370378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633ee38b-064e-470e-9a55-779e5a10e01a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:18.469777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633ee38b-064e-470e-9a55-779e5a10e01a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:18.502512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a184404-d61c-455b-b52e-df4e2e268c27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:18.503151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a184404-d61c-455b-b52e-df4e2e268c27 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:18.503433 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a184404-d61c-455b-b52e-df4e2e268c27 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:18.503703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a184404-d61c-455b-b52e-df4e2e268c27 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:18.513336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a184404-d61c-455b-b52e-df4e2e268c27 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:29:18.518947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-6a184404-d61c-455b-b52e-df4e2e268c27 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:29:18.518947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:29:18.518947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:29:18.518947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:29:18.518947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:29:18.518947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:29:18.518947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:29:18.518947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:29:18.518947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:29:18.518947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:29:18.518947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:29:18.518947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:29:18.518947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:29:18.518947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:29:18.518947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:29:18.518947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:29:18.520072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:29:18.520072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:29:18.520072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:29:18.520072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:29:18.520072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:29:18.520072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:29:18.520072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:29:18.520072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:29:18.520072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:29:18.520072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:29:18.520072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:29:18.520072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:29:18.520072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:29:18.520072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:29:18.520072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:29:18.520072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:29:18.521316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:29:18.521316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:29:18.521316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:29:18.521316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:29:18.521316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:29:18.521316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:29:18.521316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:29:18.521316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:29:18.521316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:29:18.521316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:29:18.521316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:29:18.521316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:29:18.521316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 578/1156] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:29:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:29:18.525947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d795c26d-53fd-4430-a8b4-e63f4fd0b694 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:18.528135 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:18.621593 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d795c26d-53fd-4430-a8b4-e63f4fd0b694 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:18.654682 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61aabcf0-3b1f-42b5-ad1a-9b9ac5c05697 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:18.655446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61aabcf0-3b1f-42b5-ad1a-9b9ac5c05697 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:18.655727 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61aabcf0-3b1f-42b5-ad1a-9b9ac5c05697 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:18.655999 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61aabcf0-3b1f-42b5-ad1a-9b9ac5c05697 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:18.656911 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61aabcf0-3b1f-42b5-ad1a-9b9ac5c05697 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:29:18.657214 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61aabcf0-3b1f-42b5-ad1a-9b9ac5c05697 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:18.658574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61aabcf0-3b1f-42b5-ad1a-9b9ac5c05697 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:18.666318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 579/1157] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:29:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:18.672282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a184404-d61c-455b-b52e-df4e2e268c27 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:18.696907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:18.773088 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a184404-d61c-455b-b52e-df4e2e268c27 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:18.807928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c030bd64-ef9b-4526-a17b-93275fa0d95d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:18.808552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c030bd64-ef9b-4526-a17b-93275fa0d95d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:18.808832 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c030bd64-ef9b-4526-a17b-93275fa0d95d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:18.809105 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c030bd64-ef9b-4526-a17b-93275fa0d95d None admin] PATH_INFO: `/v3/users/e0b46d861a2e431da439d1a890c98306` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:18.810218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c030bd64-ef9b-4526-a17b-93275fa0d95d None admin] RBAC: Authorizing `identity:get_user(user_id=e0b46d861a2e431da439d1a890c98306)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:18.812551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c030bd64-ef9b-4526-a17b-93275fa0d95d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:18.814161 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 579/1158] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:29:18 2022] GET /identity/v3/users/e0b46d861a2e431da439d1a890c98306 => generated 286 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:18.819916 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61aabcf0-3b1f-42b5-ad1a-9b9ac5c05697 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:18.879681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:18.916639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61aabcf0-3b1f-42b5-ad1a-9b9ac5c05697 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:18.948869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf82ab1-3a82-478f-82ff-7930a490111a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:18.949465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf82ab1-3a82-478f-82ff-7930a490111a None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:18.949745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf82ab1-3a82-478f-82ff-7930a490111a None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:18.950019 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf82ab1-3a82-478f-82ff-7930a490111a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:19.013837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf82ab1-3a82-478f-82ff-7930a490111a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:19.016068 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf82ab1-3a82-478f-82ff-7930a490111a None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:19.072404 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:19.109449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 580/1159] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:29:18 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 16 03:29:19.115426 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c030bd64-ef9b-4526-a17b-93275fa0d95d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:19.200951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:19.210384 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c030bd64-ef9b-4526-a17b-93275fa0d95d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:19.242355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043d76c3-a52a-4adf-b556-a574663e2be3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:19.242990 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043d76c3-a52a-4adf-b556-a574663e2be3 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:19.243268 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043d76c3-a52a-4adf-b556-a574663e2be3 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:19.243538 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043d76c3-a52a-4adf-b556-a574663e2be3 None admin] PATH_INFO: `/v3/projects/18dfa1ffdfd649cdb3f6c3a41a27c312` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:19.244626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043d76c3-a52a-4adf-b556-a574663e2be3 None admin] RBAC: Authorizing `identity:get_project(project_id=18dfa1ffdfd649cdb3f6c3a41a27c312)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:19.246032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043d76c3-a52a-4adf-b556-a574663e2be3 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:19.247629 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 580/1160] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:29:19 2022] GET /identity/v3/projects/18dfa1ffdfd649cdb3f6c3a41a27c312 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:19.254131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf82ab1-3a82-478f-82ff-7930a490111a None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:19.349747 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf82ab1-3a82-478f-82ff-7930a490111a None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:19.382054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f267ff2-f7c2-4d5e-8f52-daee083afc11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:19.382651 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f267ff2-f7c2-4d5e-8f52-daee083afc11 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:19.382955 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f267ff2-f7c2-4d5e-8f52-daee083afc11 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:19.383233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f267ff2-f7c2-4d5e-8f52-daee083afc11 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:19.383946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f267ff2-f7c2-4d5e-8f52-daee083afc11 None admin] RBAC: Adding query filter params (role.id=89c4b3a971224d468d33d2dc411230a2, scope.project.id=18dfa1ffdfd649cdb3f6c3a41a27c312, user.id=e0b46d861a2e431da439d1a890c98306) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:29:19.384256 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f267ff2-f7c2-4d5e-8f52-daee083afc11 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:19.385673 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f267ff2-f7c2-4d5e-8f52-daee083afc11 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:19.391760 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 581/1161] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:29:19 2022] GET /identity/v3/role_assignments?user.id=e0b46d861a2e431da439d1a890c98306&scope.project.id=18dfa1ffdfd649cdb3f6c3a41a27c312&role.id=89c4b3a971224d468d33d2dc411230a2 => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:21.553404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a978dfe-ff74-4660-bd85-aa39efac9ff3 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:21.553963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a978dfe-ff74-4660-bd85-aa39efac9ff3 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:21.553963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a978dfe-ff74-4660-bd85-aa39efac9ff3 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:21.555297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 581/1162] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:29:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:29:21.560464 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877aaaa3-276f-4810-aa6f-414c4164bf1f None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:21.560862 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877aaaa3-276f-4810-aa6f-414c4164bf1f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:21.561011 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877aaaa3-276f-4810-aa6f-414c4164bf1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:21.579782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-877aaaa3-276f-4810-aa6f-414c4164bf1f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:29:21.580895 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-877aaaa3-276f-4810-aa6f-414c4164bf1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:29:21.602602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:21.646124 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 582/1163] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:29:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:29:21.918749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a978dfe-ff74-4660-bd85-aa39efac9ff3 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:22.017428 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a978dfe-ff74-4660-bd85-aa39efac9ff3 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:22.050014 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e57bd76-c34e-429b-a32a-8cc4ee04458f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:22.050610 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e57bd76-c34e-429b-a32a-8cc4ee04458f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:22.050886 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e57bd76-c34e-429b-a32a-8cc4ee04458f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:22.051185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e57bd76-c34e-429b-a32a-8cc4ee04458f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:22.076125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:22.115751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e57bd76-c34e-429b-a32a-8cc4ee04458f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:22.117187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e57bd76-c34e-429b-a32a-8cc4ee04458f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:22.210808 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 582/1164] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:29:21 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 03:29:22.216766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877aaaa3-276f-4810-aa6f-414c4164bf1f None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:22.310930 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877aaaa3-276f-4810-aa6f-414c4164bf1f None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:22.345380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c781d5-1c2c-4230-b421-d5714b4087ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:22.346009 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c781d5-1c2c-4230-b421-d5714b4087ee None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:22.346291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c781d5-1c2c-4230-b421-d5714b4087ee None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:22.346563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c781d5-1c2c-4230-b421-d5714b4087ee None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:22.353021 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c781d5-1c2c-4230-b421-d5714b4087ee None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:22.354446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c781d5-1c2c-4230-b421-d5714b4087ee None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:22.365504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-f7c781d5-1c2c-4230-b421-d5714b4087ee None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 16 03:29:22.365504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:29:22.365504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 16 03:29:22.365504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 16 03:29:22.365504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:29:22.365504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:29:22.365504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:29:22.365504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:29:22.365504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 16 03:29:22.365504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:29:22.365504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:29:22.365504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:29:22.365504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:29:22.365504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:29:22.365504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:29:22.365504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:29:22.365504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:29:22.366679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:29:22.366679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:29:22.366679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:29:22.366679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:29:22.366679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:29:22.366679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:29:22.366679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:29:22.366679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:29:22.366679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:29:22.366679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:29:22.366679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:29:22.366679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:29:22.366679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:29:22.366679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:29:22.366679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:29:22.366679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:29:22.367589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:29:22.367589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:29:22.367589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:29:22.367589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:29:22.367589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:29:22.367589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:29:22.367589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:29:22.367589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:29:22.367589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:29:22.367589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:29:22.367589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 16 03:29:22.367589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:29:22.367589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 16 03:29:22.367589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:29:22.367589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 583/1165] 10.222.0.49 () {56 vars in 1117 bytes} [Fri Sep 16 03:29:22 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:29:22.372570 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e57bd76-c34e-429b-a32a-8cc4ee04458f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:22.429095 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:22.469347 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e57bd76-c34e-429b-a32a-8cc4ee04458f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:22.501412 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63aa96d-ed7d-4906-b6e6-fcaa45fadd8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:22.502012 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63aa96d-ed7d-4906-b6e6-fcaa45fadd8d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:22.502294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63aa96d-ed7d-4906-b6e6-fcaa45fadd8d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:22.502570 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63aa96d-ed7d-4906-b6e6-fcaa45fadd8d None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:22.503235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63aa96d-ed7d-4906-b6e6-fcaa45fadd8d None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:29:22.503541 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63aa96d-ed7d-4906-b6e6-fcaa45fadd8d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:22.504895 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63aa96d-ed7d-4906-b6e6-fcaa45fadd8d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:22.511771 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 583/1166] 10.222.0.49 () {56 vars in 1115 bytes} [Fri Sep 16 03:29:22 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:24.614677 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ca055e-6031-40ab-9f07-e534bee647a3 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:24.615230 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ca055e-6031-40ab-9f07-e534bee647a3 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:24.615230 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ca055e-6031-40ab-9f07-e534bee647a3 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:24.616504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 584/1167] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:29:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:29:24.621783 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c4ec9a-4b85-4f95-83aa-6b17ee263e01 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:24.622171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c4ec9a-4b85-4f95-83aa-6b17ee263e01 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:24.622364 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c4ec9a-4b85-4f95-83aa-6b17ee263e01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:24.641534 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-c4c4ec9a-4b85-4f95-83aa-6b17ee263e01 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:29:24.642714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c4c4ec9a-4b85-4f95-83aa-6b17ee263e01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:29:24.711648 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 584/1168] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:29:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:29:24.987689 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ca055e-6031-40ab-9f07-e534bee647a3 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:25.007329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:25.088232 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ca055e-6031-40ab-9f07-e534bee647a3 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:25.120326 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b99dab-c7a0-4c12-a5a5-54774d9438a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:25.120912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b99dab-c7a0-4c12-a5a5-54774d9438a9 None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:25.121188 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b99dab-c7a0-4c12-a5a5-54774d9438a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:25.121458 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b99dab-c7a0-4c12-a5a5-54774d9438a9 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:25.122121 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b99dab-c7a0-4c12-a5a5-54774d9438a9 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:25.123509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b99dab-c7a0-4c12-a5a5-54774d9438a9 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:25.195847 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 585/1169] 10.222.0.49 () {60 vars in 1138 bytes} [Fri Sep 16 03:29:24 2022] POST /identity/v3/domains => generated 267 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:29:27.390086 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6022f98c-ba73-4231-bd28-fd562f343eb3 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:27.390615 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6022f98c-ba73-4231-bd28-fd562f343eb3 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:27.390615 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6022f98c-ba73-4231-bd28-fd562f343eb3 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:27.392003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 585/1170] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:29:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:29:27.397200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317ecf3a-2a9c-4e71-b4df-412ab89b7c15 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:27.397645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317ecf3a-2a9c-4e71-b4df-412ab89b7c15 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:27.397747 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317ecf3a-2a9c-4e71-b4df-412ab89b7c15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:27.416594 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-317ecf3a-2a9c-4e71-b4df-412ab89b7c15 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:29:27.417769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-317ecf3a-2a9c-4e71-b4df-412ab89b7c15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:29:27.481924 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 586/1171] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:29:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:29:27.754900 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6022f98c-ba73-4231-bd28-fd562f343eb3 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:27.807398 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:27.856169 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6022f98c-ba73-4231-bd28-fd562f343eb3 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:27.890513 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f62e27-2d0c-42c9-bfe4-21ac843a76c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:27.891158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f62e27-2d0c-42c9-bfe4-21ac843a76c7 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:27.891437 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f62e27-2d0c-42c9-bfe4-21ac843a76c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:27.891709 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f62e27-2d0c-42c9-bfe4-21ac843a76c7 None admin] PATH_INFO: `/v3/domains/a6b9b265eea24875a1f249fa03d04b0f` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:27.901631 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f62e27-2d0c-42c9-bfe4-21ac843a76c7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=a6b9b265eea24875a1f249fa03d04b0f)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:27.903105 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f62e27-2d0c-42c9-bfe4-21ac843a76c7 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:27.904672 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 586/1172] 10.222.0.49 () {56 vars in 1183 bytes} [Fri Sep 16 03:29:27 2022] GET /identity/v3/domains/a6b9b265eea24875a1f249fa03d04b0f => generated 267 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:27.910937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317ecf3a-2a9c-4e71-b4df-412ab89b7c15 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:27.921309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:28.007027 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317ecf3a-2a9c-4e71-b4df-412ab89b7c15 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:28.039661 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad52edc4-592e-41e7-8779-6cc8f834f07f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:28.040263 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad52edc4-592e-41e7-8779-6cc8f834f07f None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:28.040541 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad52edc4-592e-41e7-8779-6cc8f834f07f None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:28.040810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad52edc4-592e-41e7-8779-6cc8f834f07f None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:28.041462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad52edc4-592e-41e7-8779-6cc8f834f07f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:28.042951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad52edc4-592e-41e7-8779-6cc8f834f07f None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:28.136788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 587/1173] 10.222.0.49 () {60 vars in 1141 bytes} [Fri Sep 16 03:29:27 2022] POST /identity/v3/projects => generated 361 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:29:30.312500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80370d3-b26a-4610-81a8-2e5c38d214c2 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:30.313054 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80370d3-b26a-4610-81a8-2e5c38d214c2 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:30.313054 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80370d3-b26a-4610-81a8-2e5c38d214c2 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:30.314373 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 587/1174] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:29:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:29:30.319793 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9921865-9d02-44db-8d8e-fcf2ca60be34 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:30.320218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9921865-9d02-44db-8d8e-fcf2ca60be34 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:30.320345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9921865-9d02-44db-8d8e-fcf2ca60be34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:30.338791 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-c9921865-9d02-44db-8d8e-fcf2ca60be34 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:29:30.340010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c9921865-9d02-44db-8d8e-fcf2ca60be34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:29:30.393079 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:30.403207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 588/1175] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:29:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:29:30.676287 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80370d3-b26a-4610-81a8-2e5c38d214c2 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:30.775881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80370d3-b26a-4610-81a8-2e5c38d214c2 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:30.788368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:30.808637 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d098034e-4e95-4e9b-99fd-7ba6c351ec4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:30.809250 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d098034e-4e95-4e9b-99fd-7ba6c351ec4d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:30.809525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d098034e-4e95-4e9b-99fd-7ba6c351ec4d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:30.809794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d098034e-4e95-4e9b-99fd-7ba6c351ec4d None admin] PATH_INFO: `/v3/domains/a6b9b265eea24875a1f249fa03d04b0f` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:30.810883 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d098034e-4e95-4e9b-99fd-7ba6c351ec4d None admin] RBAC: Authorizing `identity:get_domain(domain_id=a6b9b265eea24875a1f249fa03d04b0f)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:30.813175 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d098034e-4e95-4e9b-99fd-7ba6c351ec4d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:30.814694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 588/1176] 10.222.0.49 () {56 vars in 1183 bytes} [Fri Sep 16 03:29:30 2022] GET /identity/v3/domains/a6b9b265eea24875a1f249fa03d04b0f => generated 267 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:30.821376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9921865-9d02-44db-8d8e-fcf2ca60be34 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:30.919770 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9921865-9d02-44db-8d8e-fcf2ca60be34 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:30.953707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ced04c8-f322-47fc-83f2-9909d80b9be2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:30.954306 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ced04c8-f322-47fc-83f2-9909d80b9be2 None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:30.954587 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ced04c8-f322-47fc-83f2-9909d80b9be2 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:30.954860 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ced04c8-f322-47fc-83f2-9909d80b9be2 None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:30.955528 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ced04c8-f322-47fc-83f2-9909d80b9be2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:30.956976 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ced04c8-f322-47fc-83f2-9909d80b9be2 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:31.090855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 589/1177] 10.222.0.49 () {60 vars in 1132 bytes} [Fri Sep 16 03:29:30 2022] POST /identity/v3/users => generated 311 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:29:33.253372 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f364e81e-494c-4199-9677-26d76e208e3c None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:33.253954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f364e81e-494c-4199-9677-26d76e208e3c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:33.253954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f364e81e-494c-4199-9677-26d76e208e3c None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:33.255251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 589/1178] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:29:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:29:33.260484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0520e4-c9ad-42e7-ad4c-167ba825769b None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:33.260869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0520e4-c9ad-42e7-ad4c-167ba825769b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:33.261036 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0520e4-c9ad-42e7-ad4c-167ba825769b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:33.279377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-fc0520e4-c9ad-42e7-ad4c-167ba825769b None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:29:33.280855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-fc0520e4-c9ad-42e7-ad4c-167ba825769b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:29:33.325900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:33.351102 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 590/1179] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:29:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:29:33.617664 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f364e81e-494c-4199-9677-26d76e208e3c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:33.715478 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f364e81e-494c-4199-9677-26d76e208e3c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:33.747539 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5944bd5-af7e-4213-943b-b719a3ec0eb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:33.748147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5944bd5-af7e-4213-943b-b719a3ec0eb5 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:33.748425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5944bd5-af7e-4213-943b-b719a3ec0eb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:33.749249 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5944bd5-af7e-4213-943b-b719a3ec0eb5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:33.759113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5944bd5-af7e-4213-943b-b719a3ec0eb5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:29:33.763739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-b5944bd5-af7e-4213-943b-b719a3ec0eb5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:29:33.763739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:29:33.763739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:29:33.763739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:29:33.763739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:29:33.763739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:29:33.763739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:29:33.763739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:29:33.763739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:29:33.763739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:29:33.763739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:29:33.763739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:29:33.763739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:29:33.763739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:29:33.763739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:29:33.763739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:29:33.765497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:29:33.765497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:29:33.765497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:29:33.765497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:29:33.765497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:29:33.765497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:29:33.765497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:29:33.765497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:29:33.765497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:29:33.765497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:29:33.765497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:29:33.765497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:29:33.765497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:29:33.765497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:29:33.765497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:29:33.765497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:29:33.766606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:29:33.766606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:29:33.766606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:29:33.766606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:29:33.766606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:29:33.766606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:29:33.766606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:29:33.766606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:29:33.766606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:29:33.766606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:29:33.766606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:29:33.766606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:29:33.766606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 590/1180] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:29:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:29:33.771494 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0520e4-c9ad-42e7-ad4c-167ba825769b None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:33.864942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0520e4-c9ad-42e7-ad4c-167ba825769b None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:33.897942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd58777-d922-415f-a779-000163dcd2fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:33.898480 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd58777-d922-415f-a779-000163dcd2fb None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:33.898756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd58777-d922-415f-a779-000163dcd2fb None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:33.899059 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd58777-d922-415f-a779-000163dcd2fb None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:33.899705 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd58777-d922-415f-a779-000163dcd2fb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:29:33.900004 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd58777-d922-415f-a779-000163dcd2fb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:33.901365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd58777-d922-415f-a779-000163dcd2fb None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:33.909537 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 591/1181] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:29:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:33.915622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5944bd5-af7e-4213-943b-b719a3ec0eb5 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:33.957780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:34.011838 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5944bd5-af7e-4213-943b-b719a3ec0eb5 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:34.044332 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192153a7-e7ca-4e52-b65d-678d53abd040 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:34.047129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192153a7-e7ca-4e52-b65d-678d53abd040 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:34.047129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192153a7-e7ca-4e52-b65d-678d53abd040 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:34.047129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192153a7-e7ca-4e52-b65d-678d53abd040 None admin] PATH_INFO: `/v3/users/00178f3c6281466db46fdceeeeb8fd50` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:34.095420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192153a7-e7ca-4e52-b65d-678d53abd040 None admin] RBAC: Authorizing `identity:get_user(user_id=00178f3c6281466db46fdceeeeb8fd50)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:34.096848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192153a7-e7ca-4e52-b65d-678d53abd040 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:34.098451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 591/1182] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:29:33 2022] GET /identity/v3/users/00178f3c6281466db46fdceeeeb8fd50 => generated 311 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:34.104271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd58777-d922-415f-a779-000163dcd2fb None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:34.144349 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:34.202931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd58777-d922-415f-a779-000163dcd2fb None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:34.235574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bae1032-b058-4475-aca6-d16bc2d58c68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:34.236172 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bae1032-b058-4475-aca6-d16bc2d58c68 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:34.236452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bae1032-b058-4475-aca6-d16bc2d58c68 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:34.236724 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bae1032-b058-4475-aca6-d16bc2d58c68 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:34.301302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bae1032-b058-4475-aca6-d16bc2d58c68 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:34.302737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bae1032-b058-4475-aca6-d16bc2d58c68 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:34.339436 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:34.397767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 592/1183] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:29:34 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 03:29:34.403691 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192153a7-e7ca-4e52-b65d-678d53abd040 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:34.496895 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:34.501212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192153a7-e7ca-4e52-b65d-678d53abd040 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:34.534415 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ccc837-0b39-4219-981d-3e8e767f4f4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:34.535071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ccc837-0b39-4219-981d-3e8e767f4f4d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:34.535351 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ccc837-0b39-4219-981d-3e8e767f4f4d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:34.535625 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ccc837-0b39-4219-981d-3e8e767f4f4d None admin] PATH_INFO: `/v3/projects/4765317d62cc44f6916853a174ca9eb5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:34.536724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ccc837-0b39-4219-981d-3e8e767f4f4d None admin] RBAC: Authorizing `identity:get_project(project_id=4765317d62cc44f6916853a174ca9eb5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:34.538145 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ccc837-0b39-4219-981d-3e8e767f4f4d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:34.539761 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 592/1184] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:29:34 2022] GET /identity/v3/projects/4765317d62cc44f6916853a174ca9eb5 => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:34.546398 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bae1032-b058-4475-aca6-d16bc2d58c68 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:34.641969 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bae1032-b058-4475-aca6-d16bc2d58c68 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:34.675050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7408c63-cabf-422e-9e56-38e1c7d47a4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:34.675642 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7408c63-cabf-422e-9e56-38e1c7d47a4d None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:34.675922 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7408c63-cabf-422e-9e56-38e1c7d47a4d None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:34.676195 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7408c63-cabf-422e-9e56-38e1c7d47a4d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:34.676898 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7408c63-cabf-422e-9e56-38e1c7d47a4d None admin] RBAC: Adding query filter params (role.id=89c4b3a971224d468d33d2dc411230a2, scope.project.id=4765317d62cc44f6916853a174ca9eb5, user.id=00178f3c6281466db46fdceeeeb8fd50) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:29:34.677201 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7408c63-cabf-422e-9e56-38e1c7d47a4d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:34.678610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7408c63-cabf-422e-9e56-38e1c7d47a4d None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:34.683754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 593/1185] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:29:34 2022] GET /identity/v3/role_assignments?user.id=00178f3c6281466db46fdceeeeb8fd50&scope.project.id=4765317d62cc44f6916853a174ca9eb5&role.id=89c4b3a971224d468d33d2dc411230a2 => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:36.778399 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c984738-551b-4653-9905-c538e3217965 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:36.778985 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c984738-551b-4653-9905-c538e3217965 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:36.778985 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c984738-551b-4653-9905-c538e3217965 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:36.781152 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 593/1186] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:29:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:29:36.786640 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb2f921-20af-480a-a633-47ebb0e755ae None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:36.787087 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb2f921-20af-480a-a633-47ebb0e755ae None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:36.787242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb2f921-20af-480a-a633-47ebb0e755ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:36.793521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:36.807339 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-5cb2f921-20af-480a-a633-47ebb0e755ae None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:29:36.810086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-5cb2f921-20af-480a-a633-47ebb0e755ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:29:36.879862 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 594/1187] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:29:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:29:37.161082 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c984738-551b-4653-9905-c538e3217965 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:37.259050 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c984738-551b-4653-9905-c538e3217965 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:37.292036 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0570bae3-36e7-46e7-8dcd-9c74da7172dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:37.292648 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0570bae3-36e7-46e7-8dcd-9c74da7172dc None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:37.292926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0570bae3-36e7-46e7-8dcd-9c74da7172dc None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:37.293196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0570bae3-36e7-46e7-8dcd-9c74da7172dc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:37.303095 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0570bae3-36e7-46e7-8dcd-9c74da7172dc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:29:37.307780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-0570bae3-36e7-46e7-8dcd-9c74da7172dc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:29:37.307780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:29:37.307780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:29:37.307780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:29:37.307780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:29:37.307780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:29:37.307780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:29:37.307780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:29:37.307780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:29:37.307780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:29:37.307780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:29:37.307780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:29:37.307780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:29:37.307780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:29:37.307780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:29:37.307780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:29:37.308901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:29:37.308901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:29:37.308901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:29:37.308901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:29:37.308901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:29:37.308901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:29:37.308901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:29:37.308901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:29:37.308901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:29:37.308901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:29:37.308901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:29:37.308901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:29:37.308901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:29:37.308901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:29:37.308901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:29:37.308901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:29:37.310283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:29:37.310283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:29:37.310283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:29:37.310283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:29:37.310283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:29:37.310283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:29:37.310283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:29:37.310283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:29:37.310283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:29:37.310283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:29:37.310283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:29:37.310283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:29:37.310283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 594/1188] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:29:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:29:37.315702 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb2f921-20af-480a-a633-47ebb0e755ae None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:37.411618 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb2f921-20af-480a-a633-47ebb0e755ae None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:37.439958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:37.444554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a75709f-ef50-4fda-bb9e-ac5704018dab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:37.445161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a75709f-ef50-4fda-bb9e-ac5704018dab None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:37.445442 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a75709f-ef50-4fda-bb9e-ac5704018dab None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:37.445716 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a75709f-ef50-4fda-bb9e-ac5704018dab None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:37.446372 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a75709f-ef50-4fda-bb9e-ac5704018dab None admin] RBAC: Adding query filter params (name=admin) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:29:37.446714 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a75709f-ef50-4fda-bb9e-ac5704018dab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:37.448108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a75709f-ef50-4fda-bb9e-ac5704018dab None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:37.456308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 595/1189] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:29:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:37.462116 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0570bae3-36e7-46e7-8dcd-9c74da7172dc None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:37.489267 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:37.559727 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0570bae3-36e7-46e7-8dcd-9c74da7172dc None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:37.592146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3031d501-83c0-47f8-b547-5b86ceea9521 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:37.592756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3031d501-83c0-47f8-b547-5b86ceea9521 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:37.593030 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3031d501-83c0-47f8-b547-5b86ceea9521 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:37.593300 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3031d501-83c0-47f8-b547-5b86ceea9521 None admin] PATH_INFO: `/v3/users/00178f3c6281466db46fdceeeeb8fd50` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:37.594394 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3031d501-83c0-47f8-b547-5b86ceea9521 None admin] RBAC: Authorizing `identity:get_user(user_id=00178f3c6281466db46fdceeeeb8fd50)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:37.595828 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3031d501-83c0-47f8-b547-5b86ceea9521 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:37.597355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 595/1190] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:29:37 2022] GET /identity/v3/users/00178f3c6281466db46fdceeeeb8fd50 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:37.603080 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a75709f-ef50-4fda-bb9e-ac5704018dab None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:37.700415 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a75709f-ef50-4fda-bb9e-ac5704018dab None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:37.732227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fabaca-f09a-43fa-8b72-5bbb8bc64ec4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:37.733383 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fabaca-f09a-43fa-8b72-5bbb8bc64ec4 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:37.733882 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fabaca-f09a-43fa-8b72-5bbb8bc64ec4 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:37.733984 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fabaca-f09a-43fa-8b72-5bbb8bc64ec4 None admin] PATH_INFO: `/v3/projects/4765317d62cc44f6916853a174ca9eb5` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:37.735046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7fabaca-f09a-43fa-8b72-5bbb8bc64ec4 None admin] RBAC: Authorizing `identity:get_project(project_id=4765317d62cc44f6916853a174ca9eb5)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:37.736465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7fabaca-f09a-43fa-8b72-5bbb8bc64ec4 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:37.738048 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 596/1191] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:29:37 2022] GET /identity/v3/projects/4765317d62cc44f6916853a174ca9eb5 => generated 361 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:37.743969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3031d501-83c0-47f8-b547-5b86ceea9521 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:37.834180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:37.840498 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3031d501-83c0-47f8-b547-5b86ceea9521 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:37.873085 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f124ae-d522-49c6-a107-d0b451a9a19e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:37.873946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f124ae-d522-49c6-a107-d0b451a9a19e None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:37.874197 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f124ae-d522-49c6-a107-d0b451a9a19e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:37.874470 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f124ae-d522-49c6-a107-d0b451a9a19e None admin] PATH_INFO: `/v3/projects/4765317d62cc44f6916853a174ca9eb5/users/00178f3c6281466db46fdceeeeb8fd50/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:37.876369 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f124ae-d522-49c6-a107-d0b451a9a19e None admin] RBAC: Authorizing `identity:create_grant(project_id=4765317d62cc44f6916853a174ca9eb5, user_id=00178f3c6281466db46fdceeeeb8fd50, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:37.877771 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f124ae-d522-49c6-a107-d0b451a9a19e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:37.960537 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 596/1192] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:29:37 2022] PUT /identity/v3/projects/4765317d62cc44f6916853a174ca9eb5/users/00178f3c6281466db46fdceeeeb8fd50/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:29:40.081140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e36ea62-457e-4b85-b77d-387eaa1cb5ad None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:40.081719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e36ea62-457e-4b85-b77d-387eaa1cb5ad None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:40.081719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e36ea62-457e-4b85-b77d-387eaa1cb5ad None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:40.083058 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 597/1193] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:29:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:29:40.088165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1a7c89-3bce-4045-a972-83e76a3b594a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:40.088570 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1a7c89-3bce-4045-a972-83e76a3b594a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:40.088705 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1a7c89-3bce-4045-a972-83e76a3b594a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:40.106794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-0f1a7c89-3bce-4045-a972-83e76a3b594a None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:29:40.107983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0f1a7c89-3bce-4045-a972-83e76a3b594a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:29:40.170643 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 597/1194] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:29:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:29:40.441603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e36ea62-457e-4b85-b77d-387eaa1cb5ad None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:40.503041 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:40.541802 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e36ea62-457e-4b85-b77d-387eaa1cb5ad None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:40.574332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf081a8-799b-4499-bdb3-7415035ebea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:40.574977 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf081a8-799b-4499-bdb3-7415035ebea9 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:40.575267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf081a8-799b-4499-bdb3-7415035ebea9 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:40.575543 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf081a8-799b-4499-bdb3-7415035ebea9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:40.586456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf081a8-799b-4499-bdb3-7415035ebea9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:29:40.591204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-cbf081a8-799b-4499-bdb3-7415035ebea9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:29:40.591204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:29:40.591204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:29:40.591204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:29:40.591204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:29:40.591204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:29:40.591204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:29:40.591204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:29:40.591204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:29:40.591204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:29:40.591204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:29:40.591204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:29:40.591204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:29:40.591204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:29:40.591204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:29:40.591204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:29:40.592951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:29:40.592951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:29:40.592951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:29:40.592951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:29:40.592951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:29:40.592951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:29:40.592951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:29:40.592951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:29:40.592951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:29:40.592951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:29:40.592951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:29:40.592951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:29:40.592951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:29:40.592951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:29:40.592951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:29:40.592951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:29:40.594292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:29:40.594292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:29:40.594292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:29:40.594292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:29:40.594292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:29:40.594292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:29:40.594292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:29:40.594292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:29:40.594292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:29:40.594292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:29:40.594292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:29:40.594292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:29:40.594292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 598/1195] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:29:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:29:40.599071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1a7c89-3bce-4045-a972-83e76a3b594a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:40.679873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:40.694776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1a7c89-3bce-4045-a972-83e76a3b594a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:40.728462 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59be704-ab5d-4b03-ad52-e1a1f9bd3f7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:40.729060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59be704-ab5d-4b03-ad52-e1a1f9bd3f7d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:40.732973 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59be704-ab5d-4b03-ad52-e1a1f9bd3f7d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:40.732973 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59be704-ab5d-4b03-ad52-e1a1f9bd3f7d None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:40.732973 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59be704-ab5d-4b03-ad52-e1a1f9bd3f7d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:29:40.732973 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59be704-ab5d-4b03-ad52-e1a1f9bd3f7d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:40.732973 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59be704-ab5d-4b03-ad52-e1a1f9bd3f7d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:40.739935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 598/1196] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:29:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:40.745724 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf081a8-799b-4499-bdb3-7415035ebea9 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:40.842783 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf081a8-799b-4499-bdb3-7415035ebea9 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:40.864087 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:40.875141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d4073d-1677-4f10-b4bd-e835b9b75354 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:40.876009 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d4073d-1677-4f10-b4bd-e835b9b75354 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:40.876009 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d4073d-1677-4f10-b4bd-e835b9b75354 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:40.876305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d4073d-1677-4f10-b4bd-e835b9b75354 None admin] PATH_INFO: `/v3/users/00178f3c6281466db46fdceeeeb8fd50` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:40.877413 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3d4073d-1677-4f10-b4bd-e835b9b75354 None admin] RBAC: Authorizing `identity:get_user(user_id=00178f3c6281466db46fdceeeeb8fd50)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:40.878876 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3d4073d-1677-4f10-b4bd-e835b9b75354 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:40.880459 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 599/1197] 10.222.0.49 () {56 vars in 1177 bytes} [Fri Sep 16 03:29:40 2022] GET /identity/v3/users/00178f3c6281466db46fdceeeeb8fd50 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:40.886288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59be704-ab5d-4b03-ad52-e1a1f9bd3f7d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:40.985862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59be704-ab5d-4b03-ad52-e1a1f9bd3f7d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:41.019698 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164b3cad-888f-4297-900c-68af2487f340 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:41.020297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164b3cad-888f-4297-900c-68af2487f340 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:41.020577 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164b3cad-888f-4297-900c-68af2487f340 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:41.020851 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164b3cad-888f-4297-900c-68af2487f340 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:41.036554 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:41.088136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164b3cad-888f-4297-900c-68af2487f340 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:41.089564 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164b3cad-888f-4297-900c-68af2487f340 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:41.191248 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 599/1198] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:29:40 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 03:29:41.197137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d4073d-1677-4f10-b4bd-e835b9b75354 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:41.294293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d4073d-1677-4f10-b4bd-e835b9b75354 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:41.328249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a899a0c8-0267-4d6a-99e0-9a67c06daac3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:41.328886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a899a0c8-0267-4d6a-99e0-9a67c06daac3 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:41.329167 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a899a0c8-0267-4d6a-99e0-9a67c06daac3 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:41.329438 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a899a0c8-0267-4d6a-99e0-9a67c06daac3 None admin] PATH_INFO: `/v3/projects/4765317d62cc44f6916853a174ca9eb5` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:41.330595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a899a0c8-0267-4d6a-99e0-9a67c06daac3 None admin] RBAC: Authorizing `identity:get_project(project_id=4765317d62cc44f6916853a174ca9eb5)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:41.332142 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a899a0c8-0267-4d6a-99e0-9a67c06daac3 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:41.333761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 600/1199] 10.222.0.49 () {56 vars in 1186 bytes} [Fri Sep 16 03:29:41 2022] GET /identity/v3/projects/4765317d62cc44f6916853a174ca9eb5 => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:41.339820 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164b3cad-888f-4297-900c-68af2487f340 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:29:41.384909 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:41.436354 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164b3cad-888f-4297-900c-68af2487f340 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:29:41.474546 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6095ccea-98a0-4452-9354-65096b0d4f25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:29:41.475176 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6095ccea-98a0-4452-9354-65096b0d4f25 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:29:41.475462 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6095ccea-98a0-4452-9354-65096b0d4f25 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:29:41.475738 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6095ccea-98a0-4452-9354-65096b0d4f25 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:29:41.476461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6095ccea-98a0-4452-9354-65096b0d4f25 None admin] RBAC: Adding query filter params (role.id=89c4b3a971224d468d33d2dc411230a2, scope.project.id=4765317d62cc44f6916853a174ca9eb5, user.id=00178f3c6281466db46fdceeeeb8fd50) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:29:41.476797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6095ccea-98a0-4452-9354-65096b0d4f25 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:29:41.478230 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6095ccea-98a0-4452-9354-65096b0d4f25 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:29:41.484417 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 600/1200] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:29:41 2022] GET /identity/v3/role_assignments?user.id=00178f3c6281466db46fdceeeeb8fd50&scope.project.id=4765317d62cc44f6916853a174ca9eb5&role.id=89c4b3a971224d468d33d2dc411230a2 => generated 608 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:29:51.322737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=71 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:29:51.481369 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=45 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:00.768371 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94eb154f-1266-43c2-928d-f1769a819610 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:00.768989 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94eb154f-1266-43c2-928d-f1769a819610 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:00.768989 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94eb154f-1266-43c2-928d-f1769a819610 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:00.770352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 601/1201] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:31:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:31:00.775596 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad4ef70-0688-4e6e-bc72-4c0370d83a1e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:00.776042 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad4ef70-0688-4e6e-bc72-4c0370d83a1e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:00.776142 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad4ef70-0688-4e6e-bc72-4c0370d83a1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:00.794661 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-4ad4ef70-0688-4e6e-bc72-4c0370d83a1e None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:31:00.795894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4ad4ef70-0688-4e6e-bc72-4c0370d83a1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:31:00.861032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 601/1202] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:31:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:31:01.135935 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94eb154f-1266-43c2-928d-f1769a819610 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:01.235254 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94eb154f-1266-43c2-928d-f1769a819610 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:01.268706 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe643af1-bbf4-4b16-b76a-f6b307f533e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:01.269427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe643af1-bbf4-4b16-b76a-f6b307f533e7 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:01.270390 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe643af1-bbf4-4b16-b76a-f6b307f533e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:01.270840 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe643af1-bbf4-4b16-b76a-f6b307f533e7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:01.281210 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe643af1-bbf4-4b16-b76a-f6b307f533e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:01.282697 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe643af1-bbf4-4b16-b76a-f6b307f533e7 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:01.297132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-fe643af1-bbf4-4b16-b76a-f6b307f533e7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:01.297132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:01.297132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:01.297132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:01.297132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:01.297132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:01.297132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:01.297132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:01.297132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:01.297132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:01.297132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:01.297132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:01.297132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:31:01.297132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:31:01.297132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:31:01.297132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:31:01.298332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:01.298332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:01.298332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:01.298332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:01.298332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:01.298332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:01.298332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:01.298332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:01.298332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:01.298332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:01.298332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:01.298332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:01.298332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:01.298332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:01.298332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:01.298332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:01.298332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:31:01.299602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:31:01.299602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:01.299602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:31:01.299602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 602/1203] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:31:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:01.304962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad4ef70-0688-4e6e-bc72-4c0370d83a1e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:01.403222 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad4ef70-0688-4e6e-bc72-4c0370d83a1e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:01.430800 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:01.436584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd085621-60b5-4c90-a1ca-73d9a4195ec0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:01.437203 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd085621-60b5-4c90-a1ca-73d9a4195ec0 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:01.437485 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd085621-60b5-4c90-a1ca-73d9a4195ec0 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:01.437761 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd085621-60b5-4c90-a1ca-73d9a4195ec0 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:01.438401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd085621-60b5-4c90-a1ca-73d9a4195ec0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:01.438705 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd085621-60b5-4c90-a1ca-73d9a4195ec0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:01.440146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd085621-60b5-4c90-a1ca-73d9a4195ec0 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:01.452260 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 602/1204] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:31:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:01.458157 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe643af1-bbf4-4b16-b76a-f6b307f533e7 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:01.511046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:01.557531 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe643af1-bbf4-4b16-b76a-f6b307f533e7 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:01.591999 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1c5113-8ae1-40f6-8a1e-a9c67ae46153 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:01.592602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1c5113-8ae1-40f6-8a1e-a9c67ae46153 None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:01.592883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1c5113-8ae1-40f6-8a1e-a9c67ae46153 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:01.593160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1c5113-8ae1-40f6-8a1e-a9c67ae46153 None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:01.593793 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1c5113-8ae1-40f6-8a1e-a9c67ae46153 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:01.595249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1c5113-8ae1-40f6-8a1e-a9c67ae46153 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:01.690640 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 603/1205] 10.222.0.49 () {60 vars in 1132 bytes} [Fri Sep 16 03:31:01 2022] POST /identity/v3/users => generated 251 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:31:03.861471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62eee6a5-1a0f-426e-aba6-0924171296b9 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:03.862032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62eee6a5-1a0f-426e-aba6-0924171296b9 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:03.862032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62eee6a5-1a0f-426e-aba6-0924171296b9 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:03.863412 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 603/1206] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:31:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:31:03.868851 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a64dd5-f6b2-479a-aad7-d4a7e53065ff None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:03.869231 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a64dd5-f6b2-479a-aad7-d4a7e53065ff None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:03.869431 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a64dd5-f6b2-479a-aad7-d4a7e53065ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:03.887473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-a5a64dd5-f6b2-479a-aad7-d4a7e53065ff None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:31:03.894330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a5a64dd5-f6b2-479a-aad7-d4a7e53065ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:31:03.953152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 604/1207] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:31:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:31:04.220861 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62eee6a5-1a0f-426e-aba6-0924171296b9 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:04.320669 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62eee6a5-1a0f-426e-aba6-0924171296b9 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:04.356811 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6d9956-1e41-41b2-b8d9-ba62187061af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:04.357442 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6d9956-1e41-41b2-b8d9-ba62187061af None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:04.357725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6d9956-1e41-41b2-b8d9-ba62187061af None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:04.358035 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6d9956-1e41-41b2-b8d9-ba62187061af None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:04.368230 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6d9956-1e41-41b2-b8d9-ba62187061af None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:31:04.373101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-7b6d9956-1e41-41b2-b8d9-ba62187061af None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:31:04.373101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:04.373101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:04.373101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:04.373101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:04.373101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:04.373101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:04.373101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:04.373101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:04.373101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:04.373101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:04.373101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:04.373101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:31:04.373101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:31:04.373101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:31:04.373101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:31:04.374217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:31:04.374217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:31:04.374217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:31:04.374217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:31:04.374217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:04.374217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:04.374217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:04.374217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:04.374217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:04.374217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:04.374217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:04.374217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:04.374217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:04.374217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:04.374217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:04.374217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:04.375143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:04.375143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:04.375143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:04.375143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:04.375143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:31:04.375143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:31:04.375143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:31:04.375143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:31:04.375143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:31:04.375143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:31:04.375143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:31:04.375143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:04.375143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 604/1208] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:31:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:04.380776 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a64dd5-f6b2-479a-aad7-d4a7e53065ff None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:04.419311 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:04.476899 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a64dd5-f6b2-479a-aad7-d4a7e53065ff None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:04.510724 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5ef3af-f18e-4ce3-972b-85c64318025a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:04.511358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5ef3af-f18e-4ce3-972b-85c64318025a None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:04.511637 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5ef3af-f18e-4ce3-972b-85c64318025a None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:04.511904 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5ef3af-f18e-4ce3-972b-85c64318025a None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:04.512553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5ef3af-f18e-4ce3-972b-85c64318025a None admin] RBAC: Adding query filter params (name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:04.512852 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5ef3af-f18e-4ce3-972b-85c64318025a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:04.514241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5ef3af-f18e-4ce3-972b-85c64318025a None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:04.521963 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 605/1209] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:31:04 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:04.527894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6d9956-1e41-41b2-b8d9-ba62187061af None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:04.580380 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:04.628209 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6d9956-1e41-41b2-b8d9-ba62187061af None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:04.661892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2138815-632b-429d-b3e9-e8d7e9086d80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:04.663878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2138815-632b-429d-b3e9-e8d7e9086d80 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:04.663878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2138815-632b-429d-b3e9-e8d7e9086d80 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:04.663878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2138815-632b-429d-b3e9-e8d7e9086d80 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:04.672934 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2138815-632b-429d-b3e9-e8d7e9086d80 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:04.674357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2138815-632b-429d-b3e9-e8d7e9086d80 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:04.689187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-b2138815-632b-429d-b3e9-e8d7e9086d80 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:04.689187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:04.689187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:04.689187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:04.689187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:04.689187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:04.689187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:04.689187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:04.689187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:04.689187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:04.689187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:04.689187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:04.689187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:31:04.689187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:31:04.689187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:31:04.689187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:31:04.693965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:04.693965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:04.693965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:04.693965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:04.693965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:04.693965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:04.693965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:04.693965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:04.693965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:04.693965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:04.693965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:04.693965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:04.693965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:04.693965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:04.693965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:04.693965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:04.693965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:31:04.695622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:31:04.695622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:04.695622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:04.695622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 605/1210] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:31:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:04.700352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5ef3af-f18e-4ce3-972b-85c64318025a None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:04.799800 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:04.803577 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5ef3af-f18e-4ce3-972b-85c64318025a None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:04.836291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a464a6e-7330-488e-85d6-c0d74cb54a2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:04.836885 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a464a6e-7330-488e-85d6-c0d74cb54a2e None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:04.837161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a464a6e-7330-488e-85d6-c0d74cb54a2e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:04.837431 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a464a6e-7330-488e-85d6-c0d74cb54a2e None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:04.838075 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a464a6e-7330-488e-85d6-c0d74cb54a2e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:04.838376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a464a6e-7330-488e-85d6-c0d74cb54a2e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:04.839769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a464a6e-7330-488e-85d6-c0d74cb54a2e None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:04.856016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 606/1211] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:31:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:31:04.863335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2138815-632b-429d-b3e9-e8d7e9086d80 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:04.973835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2138815-632b-429d-b3e9-e8d7e9086d80 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:04.975365 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:05.007144 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8488c20d-43b1-4d11-a074-0999a86ac786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:05.007767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8488c20d-43b1-4d11-a074-0999a86ac786 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:05.008043 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8488c20d-43b1-4d11-a074-0999a86ac786 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:05.008316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8488c20d-43b1-4d11-a074-0999a86ac786 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:05.015113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8488c20d-43b1-4d11-a074-0999a86ac786 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:05.016521 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8488c20d-43b1-4d11-a074-0999a86ac786 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:05.027510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-8488c20d-43b1-4d11-a074-0999a86ac786 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 16 03:31:05.027510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:05.027510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:05.027510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:05.027510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:05.027510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:05.027510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:05.027510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:05.027510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:05.027510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:05.027510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:05.027510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:05.027510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:31:05.027510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:31:05.027510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:31:05.027510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:31:05.028675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:05.028675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:05.028675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:31:05.028675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:31:05.028675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:31:05.028675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:31:05.028675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:05.028675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:05.028675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:05.028675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:05.028675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:05.028675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:05.028675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:05.028675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:05.028675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:05.028675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:05.028675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:05.028675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:05.029886 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:05.029886 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:05.029886 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:31:05.029886 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:31:05.029886 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:31:05.029886 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:31:05.029886 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:31:05.029886 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:31:05.029886 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 16 03:31:05.029886 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:05.029886 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 606/1212] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:31:04 2022] GET /identity/v3/users/placement => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:05.034766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a464a6e-7330-488e-85d6-c0d74cb54a2e None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:05.133083 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a464a6e-7330-488e-85d6-c0d74cb54a2e None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:05.166792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c838578d-bb2d-49da-b75b-302aa84f5a83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:05.167416 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c838578d-bb2d-49da-b75b-302aa84f5a83 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:05.167693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c838578d-bb2d-49da-b75b-302aa84f5a83 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:05.167960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c838578d-bb2d-49da-b75b-302aa84f5a83 None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:05.168667 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c838578d-bb2d-49da-b75b-302aa84f5a83 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:05.169042 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c838578d-bb2d-49da-b75b-302aa84f5a83 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:05.170451 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c838578d-bb2d-49da-b75b-302aa84f5a83 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:05.187744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 607/1213] 10.222.0.49 () {56 vars in 1143 bytes} [Fri Sep 16 03:31:05 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:05.193926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8488c20d-43b1-4d11-a074-0999a86ac786 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:05.295803 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8488c20d-43b1-4d11-a074-0999a86ac786 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:05.328663 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0601b4-6765-4e1b-bd69-3b37079592b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:05.329377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0601b4-6765-4e1b-bd69-3b37079592b1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:05.329603 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0601b4-6765-4e1b-bd69-3b37079592b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:05.329878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0601b4-6765-4e1b-bd69-3b37079592b1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:05.360554 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:05.396481 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0601b4-6765-4e1b-bd69-3b37079592b1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:05.397857 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0601b4-6765-4e1b-bd69-3b37079592b1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:05.493363 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 607/1214] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:31:05 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 03:31:05.499317 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c838578d-bb2d-49da-b75b-302aa84f5a83 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:05.527219 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:05.609453 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c838578d-bb2d-49da-b75b-302aa84f5a83 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:05.643538 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72efa3f7-e5a8-4f1e-8087-fa5007ed9b1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:05.644163 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72efa3f7-e5a8-4f1e-8087-fa5007ed9b1e None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:05.645015 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72efa3f7-e5a8-4f1e-8087-fa5007ed9b1e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:05.645297 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72efa3f7-e5a8-4f1e-8087-fa5007ed9b1e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:05.655143 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72efa3f7-e5a8-4f1e-8087-fa5007ed9b1e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:05.656552 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72efa3f7-e5a8-4f1e-8087-fa5007ed9b1e None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:05.671074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-72efa3f7-e5a8-4f1e-8087-fa5007ed9b1e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:05.671074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:05.671074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:05.671074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:05.671074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:05.671074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:05.671074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:05.671074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:05.671074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:05.671074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:05.671074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:05.671074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:05.671074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:31:05.671074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:31:05.671074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:31:05.671074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:31:05.672070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:05.672070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:05.672070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:05.672070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:05.672070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:05.672070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:05.672070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:05.672070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:05.672070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:05.672070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:05.672070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:05.672070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:05.672070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:05.672070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:05.672070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:05.672070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:05.672070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:31:05.673028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:31:05.673028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:05.673028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:31:05.673028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 608/1215] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:31:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:05.678212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0601b4-6765-4e1b-bd69-3b37079592b1 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:05.740388 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:05.778236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0601b4-6765-4e1b-bd69-3b37079592b1 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:05.811004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899e7c9b-2fa2-4d89-930d-d84e7d02be63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:05.811602 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899e7c9b-2fa2-4d89-930d-d84e7d02be63 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:05.811880 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899e7c9b-2fa2-4d89-930d-d84e7d02be63 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:05.812151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899e7c9b-2fa2-4d89-930d-d84e7d02be63 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:05.812780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899e7c9b-2fa2-4d89-930d-d84e7d02be63 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:05.813081 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899e7c9b-2fa2-4d89-930d-d84e7d02be63 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:05.814432 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899e7c9b-2fa2-4d89-930d-d84e7d02be63 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:05.826694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 608/1216] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:31:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:05.833951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72efa3f7-e5a8-4f1e-8087-fa5007ed9b1e None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:05.905625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:05.936458 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72efa3f7-e5a8-4f1e-8087-fa5007ed9b1e None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:05.969325 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c466474-18b7-4a09-9a74-b7cc3baf68de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:05.969953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c466474-18b7-4a09-9a74-b7cc3baf68de None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:05.970233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c466474-18b7-4a09-9a74-b7cc3baf68de None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:05.970505 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c466474-18b7-4a09-9a74-b7cc3baf68de None admin] PATH_INFO: `/v3/projects/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:05.977210 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c466474-18b7-4a09-9a74-b7cc3baf68de None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:05.979356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c466474-18b7-4a09-9a74-b7cc3baf68de None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:05.990846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-6c466474-18b7-4a09-9a74-b7cc3baf68de None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:31:05.990846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:05.990846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:05.990846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:05.990846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:05.990846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:05.990846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:05.990846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:05.990846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:05.990846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:05.990846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:05.990846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:05.990846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:31:05.990846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:31:05.990846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:31:05.991933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:31:05.991933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:05.991933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:05.991933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:05.991933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:05.991933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:05.991933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:05.991933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:05.991933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:05.991933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:05.991933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:05.991933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:05.991933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:05.991933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:05.991933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:05.991933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:05.991933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:05.993276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:31:05.993276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:31:05.993276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:31:05.993276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:31:05.993276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:31:05.993276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:31:05.993276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:31:05.993276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:31:05.993276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 609/1217] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:31:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:05.997888 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899e7c9b-2fa2-4d89-930d-d84e7d02be63 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:06.096771 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899e7c9b-2fa2-4d89-930d-d84e7d02be63 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:06.119124 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:06.131277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d45d39a-3b25-4534-8d60-0c4668ff6e39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:06.131855 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d45d39a-3b25-4534-8d60-0c4668ff6e39 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:06.132130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d45d39a-3b25-4534-8d60-0c4668ff6e39 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:06.132400 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d45d39a-3b25-4534-8d60-0c4668ff6e39 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:06.133075 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d45d39a-3b25-4534-8d60-0c4668ff6e39 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:06.133376 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d45d39a-3b25-4534-8d60-0c4668ff6e39 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:06.134776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d45d39a-3b25-4534-8d60-0c4668ff6e39 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:06.145847 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 609/1218] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:31:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:06.152040 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c466474-18b7-4a09-9a74-b7cc3baf68de None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:06.250095 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c466474-18b7-4a09-9a74-b7cc3baf68de None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:06.278065 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:06.284010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad5b0c1-7a35-4d88-9e09-c8dde17c1e28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:06.284606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad5b0c1-7a35-4d88-9e09-c8dde17c1e28 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:06.284889 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad5b0c1-7a35-4d88-9e09-c8dde17c1e28 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:06.285715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad5b0c1-7a35-4d88-9e09-c8dde17c1e28 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:06.286530 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad5b0c1-7a35-4d88-9e09-c8dde17c1e28 None admin] RBAC: Adding query filter params (role.id=974d839067e9411aa4cb850cee33f6ae, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=176da4679fcb46148da9289fccdad5c2) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:06.286840 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad5b0c1-7a35-4d88-9e09-c8dde17c1e28 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:06.288281 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad5b0c1-7a35-4d88-9e09-c8dde17c1e28 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:06.293439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 610/1219] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:31:06 2022] GET /identity/v3/role_assignments?user.id=176da4679fcb46148da9289fccdad5c2&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=974d839067e9411aa4cb850cee33f6ae => generated 259 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:31:08.460018 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c752d9e-af4a-4a61-85f5-d46fddd3e49e None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:08.460574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c752d9e-af4a-4a61-85f5-d46fddd3e49e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:08.460574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c752d9e-af4a-4a61-85f5-d46fddd3e49e None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:08.461876 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 610/1220] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:31:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:31:08.467479 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643c8bea-5ed6-48dc-8895-59e34c91ed57 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:08.472053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643c8bea-5ed6-48dc-8895-59e34c91ed57 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:08.472053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643c8bea-5ed6-48dc-8895-59e34c91ed57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:08.487879 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-643c8bea-5ed6-48dc-8895-59e34c91ed57 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:31:08.489109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-643c8bea-5ed6-48dc-8895-59e34c91ed57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:31:08.553100 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 611/1221] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:31:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:31:08.824200 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c752d9e-af4a-4a61-85f5-d46fddd3e49e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:08.931799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c752d9e-af4a-4a61-85f5-d46fddd3e49e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:08.978526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5589bb22-5f01-431a-bfa1-6826e1cd2a4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:08.979185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5589bb22-5f01-431a-bfa1-6826e1cd2a4a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:08.979465 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5589bb22-5f01-431a-bfa1-6826e1cd2a4a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:08.979752 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5589bb22-5f01-431a-bfa1-6826e1cd2a4a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:08.992644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5589bb22-5f01-431a-bfa1-6826e1cd2a4a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:31:08.999952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-5589bb22-5f01-431a-bfa1-6826e1cd2a4a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:31:08.999952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:08.999952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:08.999952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:08.999952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:08.999952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:08.999952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:08.999952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:08.999952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:08.999952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:08.999952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:08.999952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:08.999952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:31:08.999952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:31:08.999952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:31:08.999952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:31:09.001143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:31:09.001143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:31:09.001143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:31:09.001143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:31:09.001143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:09.001143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:09.001143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:09.001143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:09.001143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:09.001143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:09.001143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:09.001143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:09.001143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:09.001143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:09.001143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:09.001143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:09.002261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:09.002261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:09.002261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:09.002261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:09.002261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:31:09.002261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:31:09.002261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:31:09.002261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:31:09.002261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:31:09.002261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:31:09.002261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:31:09.002261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:09.002261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 611/1222] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:31:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:09.009115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643c8bea-5ed6-48dc-8895-59e34c91ed57 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:09.106149 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643c8bea-5ed6-48dc-8895-59e34c91ed57 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:09.129531 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:09.139823 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4032588-2667-4cbd-8056-3dff32c61732 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:09.140425 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4032588-2667-4cbd-8056-3dff32c61732 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:09.140766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4032588-2667-4cbd-8056-3dff32c61732 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:09.141030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4032588-2667-4cbd-8056-3dff32c61732 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:09.141679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4032588-2667-4cbd-8056-3dff32c61732 None admin] RBAC: Adding query filter params (name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:09.141976 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4032588-2667-4cbd-8056-3dff32c61732 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:09.143399 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4032588-2667-4cbd-8056-3dff32c61732 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:09.150963 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 612/1223] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:31:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:09.156688 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5589bb22-5f01-431a-bfa1-6826e1cd2a4a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:09.201546 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:09.257620 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5589bb22-5f01-431a-bfa1-6826e1cd2a4a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:09.291207 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43843ac4-7924-4b3f-ac39-520f2b7db341 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:09.291838 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43843ac4-7924-4b3f-ac39-520f2b7db341 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:09.292119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43843ac4-7924-4b3f-ac39-520f2b7db341 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:09.292391 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43843ac4-7924-4b3f-ac39-520f2b7db341 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:09.303962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43843ac4-7924-4b3f-ac39-520f2b7db341 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:09.305432 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43843ac4-7924-4b3f-ac39-520f2b7db341 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:09.325320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-43843ac4-7924-4b3f-ac39-520f2b7db341 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:09.325320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:09.325320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:09.325320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:09.325320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:09.325320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:09.325320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:09.325320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:09.325320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:09.325320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:09.325320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:09.325320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:09.325320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:31:09.325320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:31:09.325320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:31:09.325320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:31:09.326526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:09.326526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:09.326526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:09.326526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:09.326526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:09.326526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:09.326526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:09.326526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:09.326526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:09.326526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:09.326526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:09.326526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:09.326526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:09.326526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:09.326526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:09.326526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:09.326526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:31:09.328353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:31:09.328353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:09.328353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:09.328353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 612/1224] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:31:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:09.332184 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4032588-2667-4cbd-8056-3dff32c61732 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:09.431268 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4032588-2667-4cbd-8056-3dff32c61732 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:09.463887 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6e1ef4-a57a-4ac6-af08-ce1447faebfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:09.464495 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e1ef4-a57a-4ac6-af08-ce1447faebfa None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:09.464778 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e1ef4-a57a-4ac6-af08-ce1447faebfa None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:09.465056 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e1ef4-a57a-4ac6-af08-ce1447faebfa None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:09.465690 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6e1ef4-a57a-4ac6-af08-ce1447faebfa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:09.465994 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6e1ef4-a57a-4ac6-af08-ce1447faebfa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:09.467394 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6e1ef4-a57a-4ac6-af08-ce1447faebfa None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:09.479287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 613/1225] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:31:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:09.485242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43843ac4-7924-4b3f-ac39-520f2b7db341 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:09.577785 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:09.584421 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43843ac4-7924-4b3f-ac39-520f2b7db341 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:09.618451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0baa633a-94b0-4840-8f5e-a3d04fb31e0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:09.619335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baa633a-94b0-4840-8f5e-a3d04fb31e0a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:09.619616 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baa633a-94b0-4840-8f5e-a3d04fb31e0a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:09.619884 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baa633a-94b0-4840-8f5e-a3d04fb31e0a None admin] PATH_INFO: `/v3/users/placement` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:09.626983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0baa633a-94b0-4840-8f5e-a3d04fb31e0a None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:09.628817 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0baa633a-94b0-4840-8f5e-a3d04fb31e0a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:09.640110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-0baa633a-94b0-4840-8f5e-a3d04fb31e0a None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 16 03:31:09.640110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:09.640110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:09.640110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:09.640110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:09.640110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:09.640110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:09.640110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:09.640110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:09.640110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:09.640110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:09.640110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:09.640110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:31:09.640110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:31:09.640110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:31:09.640110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:31:09.641087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:09.641087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:09.641087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:31:09.641087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:31:09.641087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:31:09.641087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:31:09.641087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:09.641087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:09.641087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:09.641087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:09.641087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:09.641087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:09.641087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:09.641087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:09.641087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:09.641087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:09.641087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:09.641087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:09.642151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:09.642151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:09.642151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:31:09.642151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:31:09.642151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:31:09.642151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:31:09.642151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:31:09.642151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:31:09.642151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 16 03:31:09.642151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:09.642151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 613/1226] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:31:09 2022] GET /identity/v3/users/placement => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:09.647718 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6e1ef4-a57a-4ac6-af08-ce1447faebfa None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:09.694152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:09.746049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6e1ef4-a57a-4ac6-af08-ce1447faebfa None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:09.779447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70458258-6fe9-483b-8eb2-ffa7433a4f21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:09.780046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70458258-6fe9-483b-8eb2-ffa7433a4f21 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:09.780323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70458258-6fe9-483b-8eb2-ffa7433a4f21 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:09.780627 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70458258-6fe9-483b-8eb2-ffa7433a4f21 None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:09.781341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70458258-6fe9-483b-8eb2-ffa7433a4f21 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:09.781644 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70458258-6fe9-483b-8eb2-ffa7433a4f21 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:09.783086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70458258-6fe9-483b-8eb2-ffa7433a4f21 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:09.800504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 614/1227] 10.222.0.49 () {56 vars in 1143 bytes} [Fri Sep 16 03:31:09 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:09.806331 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0baa633a-94b0-4840-8f5e-a3d04fb31e0a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:09.906814 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0baa633a-94b0-4840-8f5e-a3d04fb31e0a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:09.939881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac8452c-da78-44a0-b1d2-b8764d5e754e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:09.940501 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac8452c-da78-44a0-b1d2-b8764d5e754e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:09.940779 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac8452c-da78-44a0-b1d2-b8764d5e754e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:09.941051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac8452c-da78-44a0-b1d2-b8764d5e754e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:09.951285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac8452c-da78-44a0-b1d2-b8764d5e754e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:09.952647 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac8452c-da78-44a0-b1d2-b8764d5e754e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:09.966726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-7ac8452c-da78-44a0-b1d2-b8764d5e754e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:09.966726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:09.966726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:09.966726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:09.966726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:09.966726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:09.966726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:09.966726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:09.966726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:09.966726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:09.966726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:09.966726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:09.966726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:31:09.966726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:31:09.966726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:31:09.966726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:31:09.967942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:09.967942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:09.967942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:09.967942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:09.967942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:09.967942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:09.967942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:09.967942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:09.967942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:09.967942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:09.967942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:09.967942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:09.967942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:09.967942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:09.967942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:09.967942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:09.967942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:31:09.968863 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:31:09.968863 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:09.968863 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:09.968863 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 614/1228] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:31:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:09.974601 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70458258-6fe9-483b-8eb2-ffa7433a4f21 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:10.072879 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70458258-6fe9-483b-8eb2-ffa7433a4f21 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:10.075189 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:10.107336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ab60c3-cf03-46dd-bb15-a3ac60d17177 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:10.107947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ab60c3-cf03-46dd-bb15-a3ac60d17177 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:10.108227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ab60c3-cf03-46dd-bb15-a3ac60d17177 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:10.108499 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ab60c3-cf03-46dd-bb15-a3ac60d17177 None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:10.109134 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ab60c3-cf03-46dd-bb15-a3ac60d17177 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:10.109436 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ab60c3-cf03-46dd-bb15-a3ac60d17177 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:10.110799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ab60c3-cf03-46dd-bb15-a3ac60d17177 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:10.121850 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 615/1229] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:31:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:10.127728 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac8452c-da78-44a0-b1d2-b8764d5e754e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:10.137420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:10.226277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac8452c-da78-44a0-b1d2-b8764d5e754e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:10.259401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9f9b97-5add-461d-9136-ea0c83054521 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:10.260020 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9f9b97-5add-461d-9136-ea0c83054521 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:10.263964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9f9b97-5add-461d-9136-ea0c83054521 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:10.263964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9f9b97-5add-461d-9136-ea0c83054521 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:10.267063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9f9b97-5add-461d-9136-ea0c83054521 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:10.268480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9f9b97-5add-461d-9136-ea0c83054521 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:10.280058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-df9f9b97-5add-461d-9136-ea0c83054521 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:31:10.280058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:10.280058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:10.280058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:10.280058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:10.280058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:10.280058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:10.280058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:10.280058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:10.280058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:10.280058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:10.280058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:10.280058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:31:10.280058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:31:10.280058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:31:10.281234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:31:10.281234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:10.281234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:10.281234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:10.281234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:10.281234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:10.281234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:10.281234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:10.281234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:10.281234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:10.281234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:10.281234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:10.281234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:10.281234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:10.281234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:10.281234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:10.281234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:10.282344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:31:10.282344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:31:10.282344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:31:10.282344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:31:10.282344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:31:10.282344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:31:10.282344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:31:10.282344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:10.282344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 615/1230] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:31:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:10.287647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ab60c3-cf03-46dd-bb15-a3ac60d17177 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:10.387380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ab60c3-cf03-46dd-bb15-a3ac60d17177 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:10.420684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355f739e-3776-4688-bab0-a80b81844c7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:10.421276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355f739e-3776-4688-bab0-a80b81844c7d None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:10.421552 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355f739e-3776-4688-bab0-a80b81844c7d None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:10.421820 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355f739e-3776-4688-bab0-a80b81844c7d None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:10.422498 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355f739e-3776-4688-bab0-a80b81844c7d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:10.422794 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355f739e-3776-4688-bab0-a80b81844c7d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:10.424229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355f739e-3776-4688-bab0-a80b81844c7d None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:10.436399 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 616/1231] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:31:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:10.443647 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9f9b97-5add-461d-9136-ea0c83054521 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:10.508138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:10.541436 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9f9b97-5add-461d-9136-ea0c83054521 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:10.573694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea091e8c-0134-4596-8934-24c6244c3374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:10.574336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea091e8c-0134-4596-8934-24c6244c3374 None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:10.574614 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea091e8c-0134-4596-8934-24c6244c3374 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:10.574886 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea091e8c-0134-4596-8934-24c6244c3374 None admin] PATH_INFO: `/v3/projects/0bb7350474f349d6aee197ff35536167/users/176da4679fcb46148da9289fccdad5c2/roles/974d839067e9411aa4cb850cee33f6ae` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:10.589092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea091e8c-0134-4596-8934-24c6244c3374 None admin] RBAC: Authorizing `identity:create_grant(project_id=0bb7350474f349d6aee197ff35536167, user_id=176da4679fcb46148da9289fccdad5c2, role_id=974d839067e9411aa4cb850cee33f6ae)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:10.604089 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea091e8c-0134-4596-8934-24c6244c3374 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:10.697029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 616/1232] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:31:10 2022] PUT /identity/v3/projects/0bb7350474f349d6aee197ff35536167/users/176da4679fcb46148da9289fccdad5c2/roles/974d839067e9411aa4cb850cee33f6ae => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:31:12.836078 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f32cae-c912-4a89-9554-194e2f01acce None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:12.836635 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f32cae-c912-4a89-9554-194e2f01acce None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:12.836635 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f32cae-c912-4a89-9554-194e2f01acce None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:12.837974 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 617/1233] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:31:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:31:12.843260 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e2f11e-ca15-475d-8eb1-e58602837133 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:12.843795 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e2f11e-ca15-475d-8eb1-e58602837133 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:12.843795 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e2f11e-ca15-475d-8eb1-e58602837133 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:12.861585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-91e2f11e-ca15-475d-8eb1-e58602837133 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:31:12.862741 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-91e2f11e-ca15-475d-8eb1-e58602837133 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:31:12.927978 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 617/1234] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:31:12 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:31:13.194312 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f32cae-c912-4a89-9554-194e2f01acce None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:13.218934 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:13.290988 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f32cae-c912-4a89-9554-194e2f01acce None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:13.324694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065e38f4-a260-4b1a-83e6-9d103a19f274 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:13.325325 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065e38f4-a260-4b1a-83e6-9d103a19f274 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:13.325601 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065e38f4-a260-4b1a-83e6-9d103a19f274 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:13.325870 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065e38f4-a260-4b1a-83e6-9d103a19f274 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:13.335900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065e38f4-a260-4b1a-83e6-9d103a19f274 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:31:13.340630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-065e38f4-a260-4b1a-83e6-9d103a19f274 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:31:13.340630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:13.340630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:13.340630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:13.340630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:13.340630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:13.340630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:13.340630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:13.340630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:13.340630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:13.340630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:13.340630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:13.340630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:31:13.340630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:31:13.340630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:31:13.340630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:31:13.341755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:31:13.341755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:31:13.341755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:31:13.341755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:31:13.341755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:13.341755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:13.341755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:13.341755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:13.341755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:13.341755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:13.341755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:13.341755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:13.341755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:13.341755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:13.341755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:13.341755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:13.342861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:13.342861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:13.342861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:13.342861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:13.342861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:31:13.342861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:31:13.342861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:31:13.342861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:31:13.342861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:31:13.342861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:31:13.342861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 03:31:13.342861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:31:13.342861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 618/1235] 10.222.0.49 () {56 vars in 1102 bytes} [Fri Sep 16 03:31:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:13.348403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e2f11e-ca15-475d-8eb1-e58602837133 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:13.393073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:13.444443 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e2f11e-ca15-475d-8eb1-e58602837133 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:13.478219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a6582f-1293-4ad0-a8f5-45d5f300d623 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:13.478856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a6582f-1293-4ad0-a8f5-45d5f300d623 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:13.479193 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a6582f-1293-4ad0-a8f5-45d5f300d623 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:13.479466 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a6582f-1293-4ad0-a8f5-45d5f300d623 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:13.483168 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a6582f-1293-4ad0-a8f5-45d5f300d623 None admin] RBAC: Adding query filter params (name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:13.483168 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a6582f-1293-4ad0-a8f5-45d5f300d623 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:13.483168 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a6582f-1293-4ad0-a8f5-45d5f300d623 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:13.489279 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 618/1236] 10.222.0.49 () {56 vars in 1103 bytes} [Fri Sep 16 03:31:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:13.495213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065e38f4-a260-4b1a-83e6-9d103a19f274 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:13.577160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:13.592991 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065e38f4-a260-4b1a-83e6-9d103a19f274 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:13.625412 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4f032e-60f4-4eb7-a436-dc7d91dd5eb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:13.626028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4f032e-60f4-4eb7-a436-dc7d91dd5eb9 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:13.626308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4f032e-60f4-4eb7-a436-dc7d91dd5eb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:13.626581 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4f032e-60f4-4eb7-a436-dc7d91dd5eb9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:13.637684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4f032e-60f4-4eb7-a436-dc7d91dd5eb9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:13.639087 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4f032e-60f4-4eb7-a436-dc7d91dd5eb9 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:13.656987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-9d4f032e-60f4-4eb7-a436-dc7d91dd5eb9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:13.656987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:13.656987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:13.656987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:13.656987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:13.656987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:13.656987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:13.656987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:13.656987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:13.656987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:13.656987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:13.656987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:13.656987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:31:13.656987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:31:13.656987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:31:13.656987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:31:13.661947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:13.661947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:13.661947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:13.661947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:13.661947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:13.661947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:13.661947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:13.661947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:13.661947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:13.661947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:13.661947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:13.661947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:13.661947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:13.661947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:13.661947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:13.661947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:13.661947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:31:13.663117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:31:13.663117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:13.663117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:31:13.663117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 619/1237] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:31:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:13.665401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a6582f-1293-4ad0-a8f5-45d5f300d623 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:13.762140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a6582f-1293-4ad0-a8f5-45d5f300d623 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:13.771727 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:13.796169 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bffcaa-e711-4d28-9041-4a80655b6ccd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:13.796775 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bffcaa-e711-4d28-9041-4a80655b6ccd None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:13.797055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bffcaa-e711-4d28-9041-4a80655b6ccd None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:13.797334 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bffcaa-e711-4d28-9041-4a80655b6ccd None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:13.797970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2bffcaa-e711-4d28-9041-4a80655b6ccd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:13.798273 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2bffcaa-e711-4d28-9041-4a80655b6ccd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:13.799668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2bffcaa-e711-4d28-9041-4a80655b6ccd None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:13.811099 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 619/1238] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:31:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:13.817314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4f032e-60f4-4eb7-a436-dc7d91dd5eb9 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:13.914641 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4f032e-60f4-4eb7-a436-dc7d91dd5eb9 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:13.947406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fd47ac-bb49-411c-b45f-de512ae7d7f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:13.948070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fd47ac-bb49-411c-b45f-de512ae7d7f2 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:13.948384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fd47ac-bb49-411c-b45f-de512ae7d7f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:13.948665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fd47ac-bb49-411c-b45f-de512ae7d7f2 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:13.951471 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:13.956344 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fd47ac-bb49-411c-b45f-de512ae7d7f2 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:13.958457 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fd47ac-bb49-411c-b45f-de512ae7d7f2 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:13.969366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-d3fd47ac-bb49-411c-b45f-de512ae7d7f2 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 16 03:31:13.969366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:13.969366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:13.969366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:13.969366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:13.969366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:13.969366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:13.969366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:13.969366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:13.969366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:13.969366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:13.969366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:13.969366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:31:13.969366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:31:13.969366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:31:13.969366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:31:13.970509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:13.970509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:13.970509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:31:13.970509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:31:13.970509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:31:13.970509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:31:13.970509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:13.970509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:13.970509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:13.970509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:13.970509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:13.970509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:13.970509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:13.970509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:13.970509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:13.970509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:13.970509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:13.970509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:13.971886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:13.971886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:13.971886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:31:13.971886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:31:13.971886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:31:13.971886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:31:13.971886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:31:13.971886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:31:13.971886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 16 03:31:13.971886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:31:13.971886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 620/1239] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:31:13 2022] GET /identity/v3/users/placement => generated 87 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:13.976443 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bffcaa-e711-4d28-9041-4a80655b6ccd None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:14.077425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bffcaa-e711-4d28-9041-4a80655b6ccd None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:14.111113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51ad3ab-d07e-4b9f-881c-eb200f8c6f2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:14.111719 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51ad3ab-d07e-4b9f-881c-eb200f8c6f2d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:14.112001 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51ad3ab-d07e-4b9f-881c-eb200f8c6f2d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:14.112276 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51ad3ab-d07e-4b9f-881c-eb200f8c6f2d None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:14.113015 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51ad3ab-d07e-4b9f-881c-eb200f8c6f2d None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:14.113381 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51ad3ab-d07e-4b9f-881c-eb200f8c6f2d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:14.114877 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51ad3ab-d07e-4b9f-881c-eb200f8c6f2d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:14.132210 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 620/1240] 10.222.0.49 () {56 vars in 1143 bytes} [Fri Sep 16 03:31:13 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:14.137995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fd47ac-bb49-411c-b45f-de512ae7d7f2 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:14.241167 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fd47ac-bb49-411c-b45f-de512ae7d7f2 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:14.273773 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894beda8-bc21-48ed-a26a-db6b32f47add None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:14.274365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894beda8-bc21-48ed-a26a-db6b32f47add None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:14.274641 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894beda8-bc21-48ed-a26a-db6b32f47add None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:14.274926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894beda8-bc21-48ed-a26a-db6b32f47add None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:14.340403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:14.347284 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894beda8-bc21-48ed-a26a-db6b32f47add None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:14.348761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894beda8-bc21-48ed-a26a-db6b32f47add None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:14.445575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 621/1241] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:31:14 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 03:31:14.451458 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51ad3ab-d07e-4b9f-881c-eb200f8c6f2d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:14.478194 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:14.548894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51ad3ab-d07e-4b9f-881c-eb200f8c6f2d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:14.583767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79365cab-3f9d-4d84-90f6-33b0eb1d6053 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:14.584473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79365cab-3f9d-4d84-90f6-33b0eb1d6053 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:14.584753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79365cab-3f9d-4d84-90f6-33b0eb1d6053 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:14.585021 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79365cab-3f9d-4d84-90f6-33b0eb1d6053 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:14.594981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79365cab-3f9d-4d84-90f6-33b0eb1d6053 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:14.596348 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79365cab-3f9d-4d84-90f6-33b0eb1d6053 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:14.610919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-79365cab-3f9d-4d84-90f6-33b0eb1d6053 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:14.610919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:14.610919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:14.610919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:14.610919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:14.610919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:14.610919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:14.610919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:14.610919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:14.610919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:14.610919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:14.610919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:14.610919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:31:14.610919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:31:14.610919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:31:14.610919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:31:14.612034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:14.612034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:14.612034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:14.612034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:14.612034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:14.612034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:14.612034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:14.612034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:14.612034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:14.612034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:14.612034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:14.612034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:14.612034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:14.612034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:14.612034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:14.612034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:14.612034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:31:14.614053 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:31:14.614053 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:14.614053 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:14.614053 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 621/1242] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:31:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:14.618323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894beda8-bc21-48ed-a26a-db6b32f47add None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:14.709642 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:14.715339 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894beda8-bc21-48ed-a26a-db6b32f47add None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:14.749379 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a5467e-6a58-4b6a-a932-78ffebf29b6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:14.749977 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a5467e-6a58-4b6a-a932-78ffebf29b6e None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:14.750254 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a5467e-6a58-4b6a-a932-78ffebf29b6e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:14.750523 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a5467e-6a58-4b6a-a932-78ffebf29b6e None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:14.751179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a5467e-6a58-4b6a-a932-78ffebf29b6e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:14.751477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a5467e-6a58-4b6a-a932-78ffebf29b6e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:14.752836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a5467e-6a58-4b6a-a932-78ffebf29b6e None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:14.763608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 622/1243] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:31:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:14.769632 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79365cab-3f9d-4d84-90f6-33b0eb1d6053 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:14.850330 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:14.872209 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79365cab-3f9d-4d84-90f6-33b0eb1d6053 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:14.904625 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0eddaab-7e77-47e8-ad54-544d4e07641b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:14.905240 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eddaab-7e77-47e8-ad54-544d4e07641b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:14.905521 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eddaab-7e77-47e8-ad54-544d4e07641b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:14.905796 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eddaab-7e77-47e8-ad54-544d4e07641b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:14.913549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0eddaab-7e77-47e8-ad54-544d4e07641b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:14.915003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0eddaab-7e77-47e8-ad54-544d4e07641b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:14.926392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-e0eddaab-7e77-47e8-ad54-544d4e07641b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:31:14.926392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:14.926392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:14.926392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:14.926392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:14.926392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:14.926392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:14.926392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:14.926392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:14.926392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:14.926392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:14.926392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:14.926392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:31:14.926392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:31:14.926392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:31:14.927488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:31:14.927488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:14.927488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:14.927488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:14.927488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:14.927488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:14.927488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:14.927488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:14.927488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:14.927488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:14.927488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:14.927488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:14.927488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:14.927488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:14.927488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:14.927488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:14.927488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:14.928911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:31:14.928911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:31:14.928911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:31:14.928911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:31:14.928911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:31:14.928911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:31:14.928911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:31:14.928911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:14.928911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 622/1244] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:31:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:14.933491 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a5467e-6a58-4b6a-a932-78ffebf29b6e None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:15.031443 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a5467e-6a58-4b6a-a932-78ffebf29b6e None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:15.065439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6009af83-c69c-42d2-9fd6-0c3acf6d23ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:15.066034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6009af83-c69c-42d2-9fd6-0c3acf6d23ab None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:15.066876 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6009af83-c69c-42d2-9fd6-0c3acf6d23ab None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:15.067190 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6009af83-c69c-42d2-9fd6-0c3acf6d23ab None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:15.067871 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6009af83-c69c-42d2-9fd6-0c3acf6d23ab None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:15.068178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6009af83-c69c-42d2-9fd6-0c3acf6d23ab None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:15.069593 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6009af83-c69c-42d2-9fd6-0c3acf6d23ab None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:15.081357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 623/1245] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:31:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:15.087986 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0eddaab-7e77-47e8-ad54-544d4e07641b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:15.186425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0eddaab-7e77-47e8-ad54-544d4e07641b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:15.218761 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203d8c21-7f8a-4157-962a-88095deb78b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:15.219388 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203d8c21-7f8a-4157-962a-88095deb78b1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:15.219667 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203d8c21-7f8a-4157-962a-88095deb78b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:15.219940 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203d8c21-7f8a-4157-962a-88095deb78b1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:15.220650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203d8c21-7f8a-4157-962a-88095deb78b1 None admin] RBAC: Adding query filter params (role.id=974d839067e9411aa4cb850cee33f6ae, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=176da4679fcb46148da9289fccdad5c2) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:15.220955 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203d8c21-7f8a-4157-962a-88095deb78b1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:15.222421 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203d8c21-7f8a-4157-962a-88095deb78b1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:15.229022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 623/1246] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:31:15 2022] GET /identity/v3/role_assignments?user.id=176da4679fcb46148da9289fccdad5c2&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=974d839067e9411aa4cb850cee33f6ae => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:31:17.509122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c6ea02-7996-49a9-8398-2d1936d8a603 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:17.509670 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c6ea02-7996-49a9-8398-2d1936d8a603 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:17.509670 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c6ea02-7996-49a9-8398-2d1936d8a603 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:17.511041 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 624/1247] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:31:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:31:17.516665 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c82066-6a2e-42a6-90a2-fa575963916f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:17.517129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c82066-6a2e-42a6-90a2-fa575963916f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:17.517238 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c82066-6a2e-42a6-90a2-fa575963916f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:17.524149 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:17.537771 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-36c82066-6a2e-42a6-90a2-fa575963916f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:31:17.538992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-36c82066-6a2e-42a6-90a2-fa575963916f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:31:17.608790 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 624/1248] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:31:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:31:17.884459 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c6ea02-7996-49a9-8398-2d1936d8a603 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:17.903901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:17.985436 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c6ea02-7996-49a9-8398-2d1936d8a603 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:18.018835 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea92f30-5af4-4f49-a06e-631814d6a887 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:18.019611 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea92f30-5af4-4f49-a06e-631814d6a887 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:18.019891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea92f30-5af4-4f49-a06e-631814d6a887 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:18.020161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea92f30-5af4-4f49-a06e-631814d6a887 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:18.031176 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea92f30-5af4-4f49-a06e-631814d6a887 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:31:18.036310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-0ea92f30-5af4-4f49-a06e-631814d6a887 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:31:18.036310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:18.036310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:18.036310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:18.036310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:18.036310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:18.036310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:18.036310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:18.036310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:18.036310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:18.036310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:18.036310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:18.036310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:31:18.036310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:31:18.036310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:31:18.036310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:31:18.039067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:31:18.039067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:31:18.039067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:31:18.039067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:31:18.039067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:18.039067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:18.039067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:18.039067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:18.039067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:18.039067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:18.039067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:18.039067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:18.039067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:18.039067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:18.039067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:18.039067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:18.040141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:18.040141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:18.040141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:18.040141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:18.040141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:31:18.040141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:31:18.040141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:31:18.040141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:31:18.040141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:31:18.040141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:31:18.040141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:31:18.040141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:31:18.040141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 625/1249] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:31:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:18.043417 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c82066-6a2e-42a6-90a2-fa575963916f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:18.141231 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c82066-6a2e-42a6-90a2-fa575963916f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:18.169409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:18.174094 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22dedc40-566e-41ea-826e-9ffd325474c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:18.174699 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22dedc40-566e-41ea-826e-9ffd325474c2 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:18.175003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22dedc40-566e-41ea-826e-9ffd325474c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:18.175308 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22dedc40-566e-41ea-826e-9ffd325474c2 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:18.176505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22dedc40-566e-41ea-826e-9ffd325474c2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:18.176806 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22dedc40-566e-41ea-826e-9ffd325474c2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:18.178156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22dedc40-566e-41ea-826e-9ffd325474c2 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:18.186087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 625/1250] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:31:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:18.192687 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea92f30-5af4-4f49-a06e-631814d6a887 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:18.270132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:18.292891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea92f30-5af4-4f49-a06e-631814d6a887 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:18.327204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e9cff2-09a5-45ba-9e2f-08b23a21e180 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:18.327555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e9cff2-09a5-45ba-9e2f-08b23a21e180 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:18.327835 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e9cff2-09a5-45ba-9e2f-08b23a21e180 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:18.328105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e9cff2-09a5-45ba-9e2f-08b23a21e180 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:18.339013 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e9cff2-09a5-45ba-9e2f-08b23a21e180 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:18.345177 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e9cff2-09a5-45ba-9e2f-08b23a21e180 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:18.353966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-d4e9cff2-09a5-45ba-9e2f-08b23a21e180 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:18.353966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:18.353966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:18.353966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:18.353966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:18.353966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:18.353966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:18.353966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:18.353966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:18.353966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:18.353966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:18.353966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:18.353966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:31:18.353966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:31:18.353966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:31:18.353966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:31:18.355117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:18.355117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:18.355117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:18.355117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:18.355117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:18.355117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:18.355117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:18.355117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:18.355117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:18.355117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:18.355117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:18.355117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:18.355117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:18.355117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:18.355117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:18.355117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:18.355117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:31:18.356228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:31:18.356228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:18.356228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:31:18.356228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 626/1251] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:31:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:18.361081 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22dedc40-566e-41ea-826e-9ffd325474c2 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:18.460321 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22dedc40-566e-41ea-826e-9ffd325474c2 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:18.493143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d3f1ae-4b95-4a66-b11c-947e44fcad68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:18.493733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d3f1ae-4b95-4a66-b11c-947e44fcad68 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:18.494008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d3f1ae-4b95-4a66-b11c-947e44fcad68 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:18.494276 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d3f1ae-4b95-4a66-b11c-947e44fcad68 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:18.495013 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d3f1ae-4b95-4a66-b11c-947e44fcad68 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:18.495269 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d3f1ae-4b95-4a66-b11c-947e44fcad68 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:18.496620 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d3f1ae-4b95-4a66-b11c-947e44fcad68 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:18.507849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 626/1252] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:31:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:18.513752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e9cff2-09a5-45ba-9e2f-08b23a21e180 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:18.616668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e9cff2-09a5-45ba-9e2f-08b23a21e180 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:18.641771 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:18.649967 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933a8d22-38ed-4b61-b10b-a04a9e868a8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:18.650590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933a8d22-38ed-4b61-b10b-a04a9e868a8c None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:18.650866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933a8d22-38ed-4b61-b10b-a04a9e868a8c None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:18.651169 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933a8d22-38ed-4b61-b10b-a04a9e868a8c None admin] PATH_INFO: `/v3/users/placement` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:18.658332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933a8d22-38ed-4b61-b10b-a04a9e868a8c None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:18.660695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933a8d22-38ed-4b61-b10b-a04a9e868a8c None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:18.671424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-933a8d22-38ed-4b61-b10b-a04a9e868a8c None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 16 03:31:18.671424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:18.671424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:18.671424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:18.671424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:18.671424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:18.671424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:18.671424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:18.671424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:18.671424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:18.671424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:18.671424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:18.671424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:31:18.671424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:31:18.671424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:31:18.671424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:31:18.672546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:18.672546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:18.672546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:31:18.672546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:31:18.672546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:31:18.672546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:31:18.672546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:18.672546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:18.672546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:18.672546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:18.672546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:18.672546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:18.672546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:18.672546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:18.672546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:18.672546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:18.672546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:18.672546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:18.673671 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:18.673671 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:18.673671 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:31:18.673671 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:31:18.673671 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:31:18.673671 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:31:18.673671 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:31:18.673671 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:31:18.673671 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 16 03:31:18.673671 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:31:18.673671 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 627/1253] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:31:18 2022] GET /identity/v3/users/placement => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:18.678513 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d3f1ae-4b95-4a66-b11c-947e44fcad68 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:18.734959 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:18.778850 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d3f1ae-4b95-4a66-b11c-947e44fcad68 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:18.812518 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f26250-cf3c-4dfb-8a58-a0c9034df10d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:18.813125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f26250-cf3c-4dfb-8a58-a0c9034df10d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:18.813403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f26250-cf3c-4dfb-8a58-a0c9034df10d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:18.813676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f26250-cf3c-4dfb-8a58-a0c9034df10d None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:18.814438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f26250-cf3c-4dfb-8a58-a0c9034df10d None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:18.814748 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f26250-cf3c-4dfb-8a58-a0c9034df10d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:18.816874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f26250-cf3c-4dfb-8a58-a0c9034df10d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:18.835501 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 627/1254] 10.222.0.49 () {56 vars in 1143 bytes} [Fri Sep 16 03:31:18 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:18.841301 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933a8d22-38ed-4b61-b10b-a04a9e868a8c None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:18.939294 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933a8d22-38ed-4b61-b10b-a04a9e868a8c None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:18.971641 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f9259c-6520-4342-bf20-1f64c67159dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:18.972247 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f9259c-6520-4342-bf20-1f64c67159dd None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:18.973160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f9259c-6520-4342-bf20-1f64c67159dd None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:18.973441 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f9259c-6520-4342-bf20-1f64c67159dd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:19.027827 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:19.040318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f9259c-6520-4342-bf20-1f64c67159dd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:19.041758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f9259c-6520-4342-bf20-1f64c67159dd None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:19.134758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 628/1255] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:31:18 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 03:31:19.140603 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f26250-cf3c-4dfb-8a58-a0c9034df10d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:19.238152 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f26250-cf3c-4dfb-8a58-a0c9034df10d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:19.250994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:19.272447 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2dd6c1-c497-41b5-8029-c8a58a4d3c16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:19.273073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2dd6c1-c497-41b5-8029-c8a58a4d3c16 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:19.273354 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2dd6c1-c497-41b5-8029-c8a58a4d3c16 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:19.273627 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2dd6c1-c497-41b5-8029-c8a58a4d3c16 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:19.295664 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2dd6c1-c497-41b5-8029-c8a58a4d3c16 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:19.295664 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2dd6c1-c497-41b5-8029-c8a58a4d3c16 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:19.299640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-0a2dd6c1-c497-41b5-8029-c8a58a4d3c16 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:19.299640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:19.299640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:19.299640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:19.299640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:19.299640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:19.299640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:19.299640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:19.299640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:19.299640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:19.299640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:19.299640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:19.299640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:31:19.299640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:31:19.299640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:31:19.299640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:31:19.300799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:19.300799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:19.300799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:19.300799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:19.300799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:19.300799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:19.300799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:19.300799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:19.300799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:19.300799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:19.300799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:19.300799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:19.300799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:19.300799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:19.300799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:19.300799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:19.300799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:31:19.301734 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:31:19.301734 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:19.301734 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:19.301734 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 628/1256] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:31:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:19.306607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f9259c-6520-4342-bf20-1f64c67159dd None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:19.396385 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:19.408331 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f9259c-6520-4342-bf20-1f64c67159dd None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:19.443891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c08299a-56ff-425f-bb5a-ba7ab80dee9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:19.444495 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c08299a-56ff-425f-bb5a-ba7ab80dee9f None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:19.444778 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c08299a-56ff-425f-bb5a-ba7ab80dee9f None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:19.445055 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c08299a-56ff-425f-bb5a-ba7ab80dee9f None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:19.445691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c08299a-56ff-425f-bb5a-ba7ab80dee9f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:19.445994 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c08299a-56ff-425f-bb5a-ba7ab80dee9f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:19.447399 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c08299a-56ff-425f-bb5a-ba7ab80dee9f None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:19.458860 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 629/1257] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:31:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:31:19.464553 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2dd6c1-c497-41b5-8029-c8a58a4d3c16 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:19.568366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2dd6c1-c497-41b5-8029-c8a58a4d3c16 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:19.600431 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8026a5b5-6f4b-410b-8e99-84fc5cb6054c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:19.601055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8026a5b5-6f4b-410b-8e99-84fc5cb6054c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:19.601333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8026a5b5-6f4b-410b-8e99-84fc5cb6054c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:19.601605 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8026a5b5-6f4b-410b-8e99-84fc5cb6054c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:19.608150 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8026a5b5-6f4b-410b-8e99-84fc5cb6054c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:19.609616 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8026a5b5-6f4b-410b-8e99-84fc5cb6054c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:19.621101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-8026a5b5-6f4b-410b-8e99-84fc5cb6054c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:31:19.621101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:19.621101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:19.621101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:19.621101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:19.621101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:19.621101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:19.621101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:19.621101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:19.621101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:19.621101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:19.621101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:19.621101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:31:19.621101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:31:19.621101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:31:19.622220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:31:19.622220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:19.622220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:19.622220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:19.622220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:19.622220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:19.622220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:19.622220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:19.622220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:19.622220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:19.622220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:19.622220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:19.622220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:19.622220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:19.622220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:19.622220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:19.622220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:19.623363 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:31:19.623363 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:31:19.623363 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:31:19.623363 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:31:19.623363 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:31:19.623363 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:31:19.623363 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:31:19.623363 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:19.623363 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 629/1258] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:31:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:19.628075 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c08299a-56ff-425f-bb5a-ba7ab80dee9f None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:19.727583 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c08299a-56ff-425f-bb5a-ba7ab80dee9f None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:19.761615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a86e175-7fe9-4b83-8f5f-4a112fa7f0ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:19.762213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a86e175-7fe9-4b83-8f5f-4a112fa7f0ba None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:19.762494 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a86e175-7fe9-4b83-8f5f-4a112fa7f0ba None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:19.762766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a86e175-7fe9-4b83-8f5f-4a112fa7f0ba None admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:19.763479 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a86e175-7fe9-4b83-8f5f-4a112fa7f0ba None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:19.763778 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a86e175-7fe9-4b83-8f5f-4a112fa7f0ba None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:19.765196 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a86e175-7fe9-4b83-8f5f-4a112fa7f0ba None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:19.776918 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 630/1259] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:31:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:19.783087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8026a5b5-6f4b-410b-8e99-84fc5cb6054c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:19.810890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:19.884332 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8026a5b5-6f4b-410b-8e99-84fc5cb6054c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:19.917938 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e109bd2-bb06-4365-97d1-726739e7966c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:19.918525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e109bd2-bb06-4365-97d1-726739e7966c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:19.918802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e109bd2-bb06-4365-97d1-726739e7966c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:19.919114 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e109bd2-bb06-4365-97d1-726739e7966c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:19.919826 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e109bd2-bb06-4365-97d1-726739e7966c None admin] RBAC: Adding query filter params (role.id=89c4b3a971224d468d33d2dc411230a2, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=176da4679fcb46148da9289fccdad5c2) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:19.920131 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e109bd2-bb06-4365-97d1-726739e7966c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:19.921565 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e109bd2-bb06-4365-97d1-726739e7966c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:19.927575 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 630/1260] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:31:19 2022] GET /identity/v3/role_assignments?user.id=176da4679fcb46148da9289fccdad5c2&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=89c4b3a971224d468d33d2dc411230a2 => generated 259 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:22.141284 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef592a0-dcfb-4579-a96d-f3c4a245697a None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:22.141815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef592a0-dcfb-4579-a96d-f3c4a245697a None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:22.141815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef592a0-dcfb-4579-a96d-f3c4a245697a None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:22.143280 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 631/1261] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:31:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:31:22.148943 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0da8389-ae05-4e5a-9e59-2649a2741c8b None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:22.149386 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0da8389-ae05-4e5a-9e59-2649a2741c8b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:22.149517 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0da8389-ae05-4e5a-9e59-2649a2741c8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:22.168082 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a0da8389-ae05-4e5a-9e59-2649a2741c8b None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:31:22.169321 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a0da8389-ae05-4e5a-9e59-2649a2741c8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:31:22.234636 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 631/1262] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:31:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:31:22.506854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef592a0-dcfb-4579-a96d-f3c4a245697a None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:22.517734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:22.606057 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef592a0-dcfb-4579-a96d-f3c4a245697a None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:22.639886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c93333-1df4-4e25-b2ff-65b32351bc5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:22.640509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c93333-1df4-4e25-b2ff-65b32351bc5c None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:22.640798 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c93333-1df4-4e25-b2ff-65b32351bc5c None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:22.641060 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c93333-1df4-4e25-b2ff-65b32351bc5c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:22.651058 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c93333-1df4-4e25-b2ff-65b32351bc5c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:31:22.655865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-f7c93333-1df4-4e25-b2ff-65b32351bc5c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:31:22.655865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:22.655865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:22.655865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:22.655865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:22.655865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:22.655865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:22.655865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:22.655865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:22.655865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:22.655865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:22.655865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:22.655865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:31:22.655865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:31:22.655865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:31:22.655865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:31:22.657002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:31:22.657002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:31:22.657002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:31:22.657002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:31:22.657002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:22.657002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:22.657002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:22.657002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:22.657002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:22.657002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:22.657002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:22.657002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:22.657002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:22.657002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:22.657002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:22.657002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:22.657931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:22.657931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:22.657931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:22.657931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:22.657931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:31:22.657931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:31:22.657931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:31:22.657931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:31:22.657931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:31:22.657931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:31:22.657931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:31:22.657931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:31:22.657931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 632/1263] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:31:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:22.663442 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0da8389-ae05-4e5a-9e59-2649a2741c8b None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:22.681839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:22.761315 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0da8389-ae05-4e5a-9e59-2649a2741c8b None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:22.794940 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d092dd3e-3cd2-4b9d-a17c-618dda08cc24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:22.795550 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d092dd3e-3cd2-4b9d-a17c-618dda08cc24 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:22.795829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d092dd3e-3cd2-4b9d-a17c-618dda08cc24 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:22.796104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d092dd3e-3cd2-4b9d-a17c-618dda08cc24 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:22.796760 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d092dd3e-3cd2-4b9d-a17c-618dda08cc24 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:22.797062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d092dd3e-3cd2-4b9d-a17c-618dda08cc24 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:22.798425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d092dd3e-3cd2-4b9d-a17c-618dda08cc24 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:22.807299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 632/1264] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:31:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:22.813096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c93333-1df4-4e25-b2ff-65b32351bc5c None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:22.885641 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:22.917873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c93333-1df4-4e25-b2ff-65b32351bc5c None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:22.949990 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a10c76-e205-4327-ad0a-2350a5fd2a21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:22.950613 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a10c76-e205-4327-ad0a-2350a5fd2a21 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:22.950891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a10c76-e205-4327-ad0a-2350a5fd2a21 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:22.951193 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a10c76-e205-4327-ad0a-2350a5fd2a21 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:22.960751 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a10c76-e205-4327-ad0a-2350a5fd2a21 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:22.962165 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a10c76-e205-4327-ad0a-2350a5fd2a21 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:22.976678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-b6a10c76-e205-4327-ad0a-2350a5fd2a21 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:22.976678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:22.976678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:22.976678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:22.976678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:22.976678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:22.976678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:22.976678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:22.976678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:22.976678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:22.976678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:22.976678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:22.976678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:31:22.976678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:31:22.976678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:31:22.976678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:31:22.982010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:22.982010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:22.982010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:22.982010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:22.982010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:22.982010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:22.982010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:22.982010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:22.982010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:22.982010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:22.982010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:22.982010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:22.982010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:22.982010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:22.982010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:22.982010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:22.982010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:31:22.983179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:31:22.983179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:22.983179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:31:22.983179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 633/1265] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:31:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:22.983684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d092dd3e-3cd2-4b9d-a17c-618dda08cc24 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:23.063389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:23.081259 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d092dd3e-3cd2-4b9d-a17c-618dda08cc24 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:23.114742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c170fd-c398-4f2a-a85f-c65f99a4acab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:23.115367 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c170fd-c398-4f2a-a85f-c65f99a4acab None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:23.115640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c170fd-c398-4f2a-a85f-c65f99a4acab None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:23.115910 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c170fd-c398-4f2a-a85f-c65f99a4acab None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:23.116537 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c170fd-c398-4f2a-a85f-c65f99a4acab None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:23.116834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c170fd-c398-4f2a-a85f-c65f99a4acab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:23.118190 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c170fd-c398-4f2a-a85f-c65f99a4acab None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:23.129185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 633/1266] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:31:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:23.135127 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a10c76-e205-4327-ad0a-2350a5fd2a21 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:23.235914 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a10c76-e205-4327-ad0a-2350a5fd2a21 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:23.255583 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:23.268792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a218e83-aba1-48e3-a33e-9e0222e2f7fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:23.269939 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a218e83-aba1-48e3-a33e-9e0222e2f7fd None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:23.270214 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a218e83-aba1-48e3-a33e-9e0222e2f7fd None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:23.270484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a218e83-aba1-48e3-a33e-9e0222e2f7fd None admin] PATH_INFO: `/v3/users/placement` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:23.277679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a218e83-aba1-48e3-a33e-9e0222e2f7fd None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:23.279151 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a218e83-aba1-48e3-a33e-9e0222e2f7fd None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:23.290732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-5a218e83-aba1-48e3-a33e-9e0222e2f7fd None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 16 03:31:23.290732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:23.290732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:23.290732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:23.290732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:23.290732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:23.290732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:23.290732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:23.290732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:23.290732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:23.290732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:23.290732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:23.290732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:31:23.290732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:31:23.290732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:31:23.290732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:31:23.291865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:23.291865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:23.291865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:31:23.291865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:31:23.291865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:31:23.291865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:31:23.291865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:23.291865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:23.291865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:23.291865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:23.291865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:23.291865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:23.291865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:23.291865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:23.291865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:23.291865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:23.291865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:23.291865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:23.293293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:23.293293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:23.293293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:31:23.293293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:31:23.293293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:31:23.293293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:31:23.293293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:31:23.293293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:31:23.293293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 16 03:31:23.293293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:31:23.293293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 634/1267] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:31:23 2022] GET /identity/v3/users/placement => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:23.297651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c170fd-c398-4f2a-a85f-c65f99a4acab None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:23.397999 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c170fd-c398-4f2a-a85f-c65f99a4acab None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:23.431370 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bce5e24-0fbc-4411-ba94-ca34f106948c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:23.431969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bce5e24-0fbc-4411-ba94-ca34f106948c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:23.432248 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bce5e24-0fbc-4411-ba94-ca34f106948c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:23.432520 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bce5e24-0fbc-4411-ba94-ca34f106948c None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:23.433228 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bce5e24-0fbc-4411-ba94-ca34f106948c None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:23.433532 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bce5e24-0fbc-4411-ba94-ca34f106948c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:23.434977 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bce5e24-0fbc-4411-ba94-ca34f106948c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:23.443776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:23.451466 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 634/1268] 10.222.0.49 () {56 vars in 1143 bytes} [Fri Sep 16 03:31:23 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:23.458049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a218e83-aba1-48e3-a33e-9e0222e2f7fd None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:23.564320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a218e83-aba1-48e3-a33e-9e0222e2f7fd None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:23.599148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce68669c-7abe-40ac-a1d5-9627dfe4c4b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:23.599715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce68669c-7abe-40ac-a1d5-9627dfe4c4b0 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:23.599998 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce68669c-7abe-40ac-a1d5-9627dfe4c4b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:23.600273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce68669c-7abe-40ac-a1d5-9627dfe4c4b0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:23.610467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce68669c-7abe-40ac-a1d5-9627dfe4c4b0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:23.611933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce68669c-7abe-40ac-a1d5-9627dfe4c4b0 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:23.625719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-ce68669c-7abe-40ac-a1d5-9627dfe4c4b0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:23.625719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:23.625719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:23.625719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:23.625719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:23.625719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:23.625719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:23.625719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:23.625719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:23.625719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:23.625719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:23.625719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:23.625719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:31:23.625719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:31:23.625719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:31:23.625719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:31:23.626814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:23.626814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:23.626814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:23.626814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:23.626814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:23.626814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:23.626814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:23.626814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:23.626814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:23.626814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:23.626814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:23.626814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:23.626814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:23.626814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:23.626814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:23.626814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:23.626814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:31:23.628129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:31:23.628129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:23.628129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:31:23.628129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 635/1269] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:31:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:23.633141 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bce5e24-0fbc-4411-ba94-ca34f106948c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:23.730867 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bce5e24-0fbc-4411-ba94-ca34f106948c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:23.764751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545307b6-885c-4ad5-95b0-3b09cac95f22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:23.765398 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545307b6-885c-4ad5-95b0-3b09cac95f22 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:23.765704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545307b6-885c-4ad5-95b0-3b09cac95f22 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:23.766004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545307b6-885c-4ad5-95b0-3b09cac95f22 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:23.766630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545307b6-885c-4ad5-95b0-3b09cac95f22 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:23.766948 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545307b6-885c-4ad5-95b0-3b09cac95f22 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:23.768299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545307b6-885c-4ad5-95b0-3b09cac95f22 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:23.779760 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 635/1270] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:31:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:23.785880 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce68669c-7abe-40ac-a1d5-9627dfe4c4b0 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:23.817710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:23.882510 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce68669c-7abe-40ac-a1d5-9627dfe4c4b0 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:23.915797 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e32f4f7-3b52-43de-9212-dfe408c40ff3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:23.916425 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e32f4f7-3b52-43de-9212-dfe408c40ff3 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:23.916705 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e32f4f7-3b52-43de-9212-dfe408c40ff3 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:23.916980 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e32f4f7-3b52-43de-9212-dfe408c40ff3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:23.923506 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e32f4f7-3b52-43de-9212-dfe408c40ff3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:23.924940 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e32f4f7-3b52-43de-9212-dfe408c40ff3 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:23.936873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-2e32f4f7-3b52-43de-9212-dfe408c40ff3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:31:23.936873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:23.936873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:23.936873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:23.936873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:23.936873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:23.936873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:23.936873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:23.936873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:23.936873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:23.936873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:23.936873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:23.936873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:31:23.936873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:31:23.936873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:31:23.941016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:31:23.941016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:23.941016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:23.941016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:23.941016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:23.941016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:23.941016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:23.941016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:23.941016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:23.941016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:23.941016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:23.941016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:23.941016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:23.941016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:23.941016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:23.941016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:23.941016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:23.942685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:31:23.942685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:31:23.942685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:31:23.942685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:31:23.942685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:31:23.942685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:31:23.942685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:31:23.942685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:31:23.942685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 636/1271] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:31:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:23.944562 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545307b6-885c-4ad5-95b0-3b09cac95f22 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:24.000712 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:24.042366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545307b6-885c-4ad5-95b0-3b09cac95f22 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:24.075588 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2039bd4c-cf34-480a-856e-2ba84dd6b73e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:24.076189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2039bd4c-cf34-480a-856e-2ba84dd6b73e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:24.076466 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2039bd4c-cf34-480a-856e-2ba84dd6b73e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:24.076740 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2039bd4c-cf34-480a-856e-2ba84dd6b73e None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:24.077418 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2039bd4c-cf34-480a-856e-2ba84dd6b73e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:24.077805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2039bd4c-cf34-480a-856e-2ba84dd6b73e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:24.079251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2039bd4c-cf34-480a-856e-2ba84dd6b73e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:24.091137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 636/1272] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:31:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:24.097118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e32f4f7-3b52-43de-9212-dfe408c40ff3 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:24.190392 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:24.199678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e32f4f7-3b52-43de-9212-dfe408c40ff3 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:24.232265 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e317e62a-7c40-4933-b994-b853e64d0968 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:24.232918 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e317e62a-7c40-4933-b994-b853e64d0968 None admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:24.233199 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e317e62a-7c40-4933-b994-b853e64d0968 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:24.233474 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e317e62a-7c40-4933-b994-b853e64d0968 None admin] PATH_INFO: `/v3/projects/0bb7350474f349d6aee197ff35536167/users/176da4679fcb46148da9289fccdad5c2/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:24.235432 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e317e62a-7c40-4933-b994-b853e64d0968 None admin] RBAC: Authorizing `identity:create_grant(project_id=0bb7350474f349d6aee197ff35536167, user_id=176da4679fcb46148da9289fccdad5c2, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:24.236861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e317e62a-7c40-4933-b994-b853e64d0968 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:24.428665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 637/1273] 10.222.0.49 () {58 vars in 1439 bytes} [Fri Sep 16 03:31:24 2022] PUT /identity/v3/projects/0bb7350474f349d6aee197ff35536167/users/176da4679fcb46148da9289fccdad5c2/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:31:26.613456 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc8ec68-5866-4b5f-99a2-dfb76f838c51 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:26.613983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc8ec68-5866-4b5f-99a2-dfb76f838c51 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:26.613983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc8ec68-5866-4b5f-99a2-dfb76f838c51 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:26.615246 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 637/1274] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:31:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:31:26.620486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb02464d-1cb4-44b0-9b07-1e7c2ebcfc9d None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:26.620913 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb02464d-1cb4-44b0-9b07-1e7c2ebcfc9d None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:26.621036 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb02464d-1cb4-44b0-9b07-1e7c2ebcfc9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:26.639440 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-fb02464d-1cb4-44b0-9b07-1e7c2ebcfc9d None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:31:26.640608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-fb02464d-1cb4-44b0-9b07-1e7c2ebcfc9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:31:26.705048 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 638/1275] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:31:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:31:26.985817 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc8ec68-5866-4b5f-99a2-dfb76f838c51 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:27.085077 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc8ec68-5866-4b5f-99a2-dfb76f838c51 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:27.102170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:27.118381 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcbdcc0-cb87-4a36-a01a-7f049ab06a1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:27.119041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcbdcc0-cb87-4a36-a01a-7f049ab06a1d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:27.119327 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcbdcc0-cb87-4a36-a01a-7f049ab06a1d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:27.119603 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcbdcc0-cb87-4a36-a01a-7f049ab06a1d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:27.129728 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcbdcc0-cb87-4a36-a01a-7f049ab06a1d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 03:31:27.134454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-cbcbdcc0-cb87-4a36-a01a-7f049ab06a1d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:31:27.134454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:27.134454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:27.134454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:27.134454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:27.134454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:27.134454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:27.134454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:27.134454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:27.134454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:27.134454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:27.134454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:27.134454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 03:31:27.134454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 03:31:27.134454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 03:31:27.134454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 03:31:27.135777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 03:31:27.135777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 03:31:27.135777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 03:31:27.135777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 03:31:27.135777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:27.135777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:27.135777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:27.135777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:27.135777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:27.135777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:27.135777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:27.135777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:27.135777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:27.135777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:27.135777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:27.135777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:27.136999 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:27.136999 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:27.136999 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:27.136999 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:27.136999 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 03:31:27.136999 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 03:31:27.136999 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 03:31:27.136999 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 03:31:27.136999 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 03:31:27.136999 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 03:31:27.136999 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 03:31:27.136999 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:27.136999 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 638/1276] 10.222.0.49 () {56 vars in 1096 bytes} [Fri Sep 16 03:31:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:27.141526 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb02464d-1cb4-44b0-9b07-1e7c2ebcfc9d None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:27.231986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:27.249426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb02464d-1cb4-44b0-9b07-1e7c2ebcfc9d None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:27.282081 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4335b09-f118-4836-baf0-bae76e1aa999 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:27.289118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4335b09-f118-4836-baf0-bae76e1aa999 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:27.289118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4335b09-f118-4836-baf0-bae76e1aa999 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:27.289118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4335b09-f118-4836-baf0-bae76e1aa999 None admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:27.289118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4335b09-f118-4836-baf0-bae76e1aa999 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:27.289118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4335b09-f118-4836-baf0-bae76e1aa999 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:27.289118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4335b09-f118-4836-baf0-bae76e1aa999 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:27.294365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 639/1277] 10.222.0.49 () {56 vars in 1099 bytes} [Fri Sep 16 03:31:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:27.300230 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcbdcc0-cb87-4a36-a01a-7f049ab06a1d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:27.400728 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcbdcc0-cb87-4a36-a01a-7f049ab06a1d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:27.434460 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ced7f7-ea0f-430f-9f80-1b4dfc875b96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:27.435110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ced7f7-ea0f-430f-9f80-1b4dfc875b96 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:27.435388 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ced7f7-ea0f-430f-9f80-1b4dfc875b96 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:27.435656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ced7f7-ea0f-430f-9f80-1b4dfc875b96 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:27.448068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ced7f7-ea0f-430f-9f80-1b4dfc875b96 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:27.449509 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ced7f7-ea0f-430f-9f80-1b4dfc875b96 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:27.464382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-44ced7f7-ea0f-430f-9f80-1b4dfc875b96 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:27.464382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:27.464382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:27.464382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:27.464382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:27.464382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:27.464382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:27.464382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:27.464382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:27.464382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:27.464382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:27.464382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:27.464382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:31:27.464382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:31:27.464382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:31:27.464382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:31:27.465551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:27.465551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:27.465551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:27.465551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:27.465551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:27.465551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:27.465551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:27.465551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:27.465551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:27.465551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:27.465551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:27.465551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:27.465551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:27.465551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:27.465551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:27.465551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:27.465551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:31:27.466683 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:31:27.466683 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:27.466683 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:27.466683 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 639/1278] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:31:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:27.471313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4335b09-f118-4836-baf0-bae76e1aa999 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:27.571380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4335b09-f118-4836-baf0-bae76e1aa999 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:27.604078 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d12ae03-4672-4b2e-a6b2-91bcc8f0fa7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:27.604678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d12ae03-4672-4b2e-a6b2-91bcc8f0fa7c None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:27.604963 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d12ae03-4672-4b2e-a6b2-91bcc8f0fa7c None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:27.605240 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d12ae03-4672-4b2e-a6b2-91bcc8f0fa7c None admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:27.605875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d12ae03-4672-4b2e-a6b2-91bcc8f0fa7c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:27.606177 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d12ae03-4672-4b2e-a6b2-91bcc8f0fa7c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:27.607568 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d12ae03-4672-4b2e-a6b2-91bcc8f0fa7c None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:27.618855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 640/1279] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:31:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:27.624663 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ced7f7-ea0f-430f-9f80-1b4dfc875b96 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:27.656427 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:27.725813 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ced7f7-ea0f-430f-9f80-1b4dfc875b96 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:27.761758 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd99c5c-04de-44c7-823c-2a11d4176e9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:27.762384 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd99c5c-04de-44c7-823c-2a11d4176e9d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:27.762666 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd99c5c-04de-44c7-823c-2a11d4176e9d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:27.762975 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd99c5c-04de-44c7-823c-2a11d4176e9d None admin] PATH_INFO: `/v3/users/placement` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:27.769993 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd99c5c-04de-44c7-823c-2a11d4176e9d None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:27.771459 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd99c5c-04de-44c7-823c-2a11d4176e9d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:27.782732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-3fd99c5c-04de-44c7-823c-2a11d4176e9d None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 16 03:31:27.782732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:27.782732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:27.782732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:27.782732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:27.782732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:27.782732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:27.782732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:27.782732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:27.782732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:27.782732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:27.782732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:27.782732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 03:31:27.782732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 03:31:27.782732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 03:31:27.782732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 03:31:27.784136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:27.784136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:27.784136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 03:31:27.784136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:31:27.784136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 03:31:27.784136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 03:31:27.784136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:27.784136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:27.784136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:27.784136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:27.784136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:27.784136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:27.784136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:27.784136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:27.784136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:27.784136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:27.784136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:27.784136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:27.785355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:27.785355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:27.785355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 03:31:27.785355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 03:31:27.785355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 03:31:27.785355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 03:31:27.785355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 03:31:27.785355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 03:31:27.785355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 16 03:31:27.785355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:27.785355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 640/1280] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:31:27 2022] GET /identity/v3/users/placement => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 03:31:27.790207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d12ae03-4672-4b2e-a6b2-91bcc8f0fa7c None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:27.794409 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:27.888555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d12ae03-4672-4b2e-a6b2-91bcc8f0fa7c None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:27.923313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4740a9c1-ea8f-4870-8bbf-cc04313c40f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:27.923955 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4740a9c1-ea8f-4870-8bbf-cc04313c40f4 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:27.924260 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4740a9c1-ea8f-4870-8bbf-cc04313c40f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:27.924528 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4740a9c1-ea8f-4870-8bbf-cc04313c40f4 None admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:27.925741 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4740a9c1-ea8f-4870-8bbf-cc04313c40f4 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:27.926080 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4740a9c1-ea8f-4870-8bbf-cc04313c40f4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:27.927836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4740a9c1-ea8f-4870-8bbf-cc04313c40f4 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:27.945601 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 641/1281] 10.222.0.49 () {56 vars in 1143 bytes} [Fri Sep 16 03:31:27 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:27.951380 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd99c5c-04de-44c7-823c-2a11d4176e9d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:28.041061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:28.050132 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd99c5c-04de-44c7-823c-2a11d4176e9d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:28.083130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fb113b-d404-417d-902b-6b8f34ce0d98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:28.083800 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fb113b-d404-417d-902b-6b8f34ce0d98 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:28.084075 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fb113b-d404-417d-902b-6b8f34ce0d98 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:28.084342 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fb113b-d404-417d-902b-6b8f34ce0d98 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:28.151855 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fb113b-d404-417d-902b-6b8f34ce0d98 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:28.153275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fb113b-d404-417d-902b-6b8f34ce0d98 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:28.239809 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:28.249550 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 641/1282] 10.222.0.49 () {58 vars in 1282 bytes} [Fri Sep 16 03:31:27 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 03:31:28.255633 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4740a9c1-ea8f-4870-8bbf-cc04313c40f4 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:28.317854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:28.355527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4740a9c1-ea8f-4870-8bbf-cc04313c40f4 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:28.388257 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6064830e-25bf-4554-aa17-de14e596561e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:28.388873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6064830e-25bf-4554-aa17-de14e596561e None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:28.389149 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6064830e-25bf-4554-aa17-de14e596561e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:28.389419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6064830e-25bf-4554-aa17-de14e596561e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:28.399711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6064830e-25bf-4554-aa17-de14e596561e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:28.401178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6064830e-25bf-4554-aa17-de14e596561e None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:28.432462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-6064830e-25bf-4554-aa17-de14e596561e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:28.432462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:28.432462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:28.432462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:28.432462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:28.432462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:28.432462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:28.432462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:28.432462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:28.432462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:28.432462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:28.432462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:28.432462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 03:31:28.432462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 03:31:28.432462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 03:31:28.432462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 03:31:28.433699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:28.433699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:28.433699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:28.433699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:28.433699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:28.433699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:28.433699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:28.433699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:28.433699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:28.433699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:28.433699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:28.433699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:28.433699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:28.433699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:28.433699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:28.433699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:28.433699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 03:31:28.434828 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 03:31:28.434828 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 03:31:28.434828 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:31:28.434828 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 642/1283] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:31:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:28.435141 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fb113b-d404-417d-902b-6b8f34ce0d98 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:28.520544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fb113b-d404-417d-902b-6b8f34ce0d98 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:28.553743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f5a79f-7c35-4ca7-8528-1fd0c0901459 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:28.554344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f5a79f-7c35-4ca7-8528-1fd0c0901459 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:28.554624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f5a79f-7c35-4ca7-8528-1fd0c0901459 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:28.554894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f5a79f-7c35-4ca7-8528-1fd0c0901459 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:28.555554 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f5a79f-7c35-4ca7-8528-1fd0c0901459 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:28.555854 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f5a79f-7c35-4ca7-8528-1fd0c0901459 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:28.557206 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f5a79f-7c35-4ca7-8528-1fd0c0901459 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:28.568870 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 642/1284] 10.222.0.49 () {56 vars in 1109 bytes} [Fri Sep 16 03:31:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:28.574518 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6064830e-25bf-4554-aa17-de14e596561e None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:28.670473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6064830e-25bf-4554-aa17-de14e596561e None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:28.684180 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:28.704530 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23171944-7074-4845-9e85-0afea990a37e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:28.705156 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23171944-7074-4845-9e85-0afea990a37e None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:28.705437 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23171944-7074-4845-9e85-0afea990a37e None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:28.705710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23171944-7074-4845-9e85-0afea990a37e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:28.712445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23171944-7074-4845-9e85-0afea990a37e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:28.713891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23171944-7074-4845-9e85-0afea990a37e None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:28.725556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application [None req-23171944-7074-4845-9e85-0afea990a37e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:31:28.725556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:28.725556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:28.725556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:28.725556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:28.725556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:28.725556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:28.725556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:28.725556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:28.725556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:28.725556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:28.725556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:28.725556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 03:31:28.725556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 03:31:28.725556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 03:31:28.726684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 03:31:28.726684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:28.726684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:28.726684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:28.726684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:28.726684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:28.726684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:28.726684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:28.726684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:28.726684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:28.726684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:28.726684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:28.726684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:28.726684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:28.726684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:28.726684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:28.726684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:28.727668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 03:31:28.727668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 03:31:28.727668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 03:31:28.727668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 03:31:28.727668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 03:31:28.727668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 03:31:28.727668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 03:31:28.727668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: ERROR keystone.server.flask.application Sep 16 03:31:28.728147 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 643/1285] 10.222.0.49 () {56 vars in 1111 bytes} [Fri Sep 16 03:31:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:28.732524 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f5a79f-7c35-4ca7-8528-1fd0c0901459 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:28.781661 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:28.829894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f5a79f-7c35-4ca7-8528-1fd0c0901459 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:28.861593 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7348c22-42b9-480a-86b2-20b74b7e2564 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:28.862189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7348c22-42b9-480a-86b2-20b74b7e2564 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:28.862464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7348c22-42b9-480a-86b2-20b74b7e2564 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:28.862844 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7348c22-42b9-480a-86b2-20b74b7e2564 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:28.864080 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7348c22-42b9-480a-86b2-20b74b7e2564 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:28.864380 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7348c22-42b9-480a-86b2-20b74b7e2564 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:28.866187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7348c22-42b9-480a-86b2-20b74b7e2564 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:28.878312 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 643/1286] 10.222.0.49 () {56 vars in 1148 bytes} [Fri Sep 16 03:31:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:28.885573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23171944-7074-4845-9e85-0afea990a37e None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:28.989596 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23171944-7074-4845-9e85-0afea990a37e None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:29.021674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c33996d-def5-4015-af0d-6a506cb9146f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:29.022264 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c33996d-def5-4015-af0d-6a506cb9146f None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:29.022541 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c33996d-def5-4015-af0d-6a506cb9146f None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:29.022808 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c33996d-def5-4015-af0d-6a506cb9146f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:29.023547 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c33996d-def5-4015-af0d-6a506cb9146f None admin] RBAC: Adding query filter params (role.id=89c4b3a971224d468d33d2dc411230a2, scope.project.id=0bb7350474f349d6aee197ff35536167, user.id=176da4679fcb46148da9289fccdad5c2) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:29.023848 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c33996d-def5-4015-af0d-6a506cb9146f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:29.025264 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c33996d-def5-4015-af0d-6a506cb9146f None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:29.030956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 644/1287] 10.222.0.49 () {56 vars in 1374 bytes} [Fri Sep 16 03:31:28 2022] GET /identity/v3/role_assignments?user.id=176da4679fcb46148da9289fccdad5c2&scope.project.id=0bb7350474f349d6aee197ff35536167&role.id=89c4b3a971224d468d33d2dc411230a2 => generated 608 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:31.248276 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e1edcc-2ffe-443a-b590-e07a1c5469f0 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:31.248866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e1edcc-2ffe-443a-b590-e07a1c5469f0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:31.248866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e1edcc-2ffe-443a-b590-e07a1c5469f0 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:31.250131 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 644/1288] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:31:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:31:31.255465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e8a28b-13d0-4742-a1d5-ae670d0072bb None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:31.263191 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e8a28b-13d0-4742-a1d5-ae670d0072bb None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:31.263191 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e8a28b-13d0-4742-a1d5-ae670d0072bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:31.273828 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-60e8a28b-13d0-4742-a1d5-ae670d0072bb None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:31:31.275026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-60e8a28b-13d0-4742-a1d5-ae670d0072bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:31:31.304429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:31.339971 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 645/1289] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:31:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:31:31.605579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e1edcc-2ffe-443a-b590-e07a1c5469f0 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:31.703894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e1edcc-2ffe-443a-b590-e07a1c5469f0 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:31.715219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:31.737597 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55b253c-6ea3-465d-8607-aeb53ab1a7bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:31.739889 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55b253c-6ea3-465d-8607-aeb53ab1a7bd None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:31.739889 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55b253c-6ea3-465d-8607-aeb53ab1a7bd None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:31.739889 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55b253c-6ea3-465d-8607-aeb53ab1a7bd None admin] PATH_INFO: `/v3/services/placement` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:31.739889 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55b253c-6ea3-465d-8607-aeb53ab1a7bd None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:31.740891 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55b253c-6ea3-465d-8607-aeb53ab1a7bd None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:31.749261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-a55b253c-6ea3-465d-8607-aeb53ab1a7bd None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 16 03:31:31.749261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:31.749261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:31:31.749261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:31:31.749261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:31:31.749261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:31:31.749261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:31:31.749261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:31:31.749261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 16 03:31:31.749261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:31.749261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:31:31.749261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:31.749261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:31.749261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:31.749261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:31.749261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:31.749261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:31.750530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:31.750530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:31.750530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:31.750530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:31.750530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:31.750530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:31.750530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:31:31.750530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:31:31.750530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:31:31.750530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:31:31.750530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:31.750530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:31.750530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:31.750530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:31.750530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:31.750530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:31.751692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:31.751692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:31.751692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:31.751692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:31.751692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:31.751692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:31.751692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:31.751692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:31.751692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:31.751692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:31.751692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:31:31.751692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:31:31.751692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 16 03:31:31.751692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:31.751692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 645/1290] 10.222.0.49 () {56 vars in 1117 bytes} [Fri Sep 16 03:31:31 2022] GET /identity/v3/services/placement => generated 90 bytes in 145 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:31.756419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e8a28b-13d0-4742-a1d5-ae670d0072bb None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:31.860043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e8a28b-13d0-4742-a1d5-ae670d0072bb None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:31.893073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a18c4bd-9aa7-40c0-ae49-d611f0481c9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:31.896179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a18c4bd-9aa7-40c0-ae49-d611f0481c9a None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:31.896179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a18c4bd-9aa7-40c0-ae49-d611f0481c9a None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:31.896179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a18c4bd-9aa7-40c0-ae49-d611f0481c9a None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:31.896179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a18c4bd-9aa7-40c0-ae49-d611f0481c9a None admin] RBAC: Adding query filter params (name=placement) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:31.896179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a18c4bd-9aa7-40c0-ae49-d611f0481c9a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:31.896602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a18c4bd-9aa7-40c0-ae49-d611f0481c9a None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:31.901240 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 646/1291] 10.222.0.49 () {56 vars in 1116 bytes} [Fri Sep 16 03:31:31 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:31.907117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55b253c-6ea3-465d-8607-aeb53ab1a7bd None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:32.014097 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55b253c-6ea3-465d-8607-aeb53ab1a7bd None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:32.046684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86827b6b-2854-4c1c-9b16-7c721279cb7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:32.047312 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86827b6b-2854-4c1c-9b16-7c721279cb7d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:32.047594 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86827b6b-2854-4c1c-9b16-7c721279cb7d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:32.047868 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86827b6b-2854-4c1c-9b16-7c721279cb7d None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:32.048490 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86827b6b-2854-4c1c-9b16-7c721279cb7d None admin] RBAC: Adding query filter params (type=placement) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:32.048790 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86827b6b-2854-4c1c-9b16-7c721279cb7d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:32.050135 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86827b6b-2854-4c1c-9b16-7c721279cb7d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:32.055013 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 646/1292] 10.222.0.49 () {56 vars in 1116 bytes} [Fri Sep 16 03:31:31 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:34.208420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ceb1c1a-4b35-4998-8b5c-5e36a0604ca3 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:34.209042 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ceb1c1a-4b35-4998-8b5c-5e36a0604ca3 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:34.209042 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ceb1c1a-4b35-4998-8b5c-5e36a0604ca3 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:34.210287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 647/1293] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:31:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:31:34.215552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e950bf5c-aeaf-4e42-afd0-d85e5e0bb5a0 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:34.216000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e950bf5c-aeaf-4e42-afd0-d85e5e0bb5a0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:34.216145 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e950bf5c-aeaf-4e42-afd0-d85e5e0bb5a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:34.234564 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e950bf5c-aeaf-4e42-afd0-d85e5e0bb5a0 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:31:34.235797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e950bf5c-aeaf-4e42-afd0-d85e5e0bb5a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:31:34.264516 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:34.301822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 647/1294] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:31:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:31:34.569713 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ceb1c1a-4b35-4998-8b5c-5e36a0604ca3 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:34.603430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:34.675797 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ceb1c1a-4b35-4998-8b5c-5e36a0604ca3 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:34.710785 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c928882-fdcd-4dfc-90a0-11c7b00f63d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:34.711419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c928882-fdcd-4dfc-90a0-11c7b00f63d1 None admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:34.711694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c928882-fdcd-4dfc-90a0-11c7b00f63d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:34.711965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c928882-fdcd-4dfc-90a0-11c7b00f63d1 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:34.712607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c928882-fdcd-4dfc-90a0-11c7b00f63d1 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:34.713972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c928882-fdcd-4dfc-90a0-11c7b00f63d1 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:34.806086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 648/1295] 10.222.0.49 () {60 vars in 1141 bytes} [Fri Sep 16 03:31:34 2022] POST /identity/v3/services => generated 246 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:31:36.950873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049a7c22-e575-48cb-b915-c9dd228cd2fe None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:36.951467 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049a7c22-e575-48cb-b915-c9dd228cd2fe None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:36.951467 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049a7c22-e575-48cb-b915-c9dd228cd2fe None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:36.952810 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 648/1296] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:31:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:31:36.958291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06e7273-8a7c-4301-ab49-3e19f5672fbe None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:36.958749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06e7273-8a7c-4301-ab49-3e19f5672fbe None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:36.958875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06e7273-8a7c-4301-ab49-3e19f5672fbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:36.977508 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-b06e7273-8a7c-4301-ab49-3e19f5672fbe None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:31:36.978680 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b06e7273-8a7c-4301-ab49-3e19f5672fbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:31:37.051528 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 649/1297] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:31:36 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:31:37.334680 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049a7c22-e575-48cb-b915-c9dd228cd2fe None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:37.433847 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049a7c22-e575-48cb-b915-c9dd228cd2fe None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:37.466615 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a3898c-360e-4b54-8cc7-2be6a1a4e8b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:37.467259 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a3898c-360e-4b54-8cc7-2be6a1a4e8b2 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:37.467540 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a3898c-360e-4b54-8cc7-2be6a1a4e8b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:37.467812 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a3898c-360e-4b54-8cc7-2be6a1a4e8b2 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:37.468389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a3898c-360e-4b54-8cc7-2be6a1a4e8b2 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:37.469755 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a3898c-360e-4b54-8cc7-2be6a1a4e8b2 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:37.478471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-d9a3898c-360e-4b54-8cc7-2be6a1a4e8b2 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 16 03:31:37.478471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:37.478471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:31:37.478471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:31:37.478471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:31:37.478471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:31:37.478471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:31:37.478471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:31:37.478471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 16 03:31:37.478471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:37.478471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:31:37.478471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:37.478471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:37.478471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:37.478471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:37.478471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:37.478471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:37.479762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:37.479762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:37.479762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:37.479762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:37.479762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:37.479762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:37.479762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:31:37.479762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:31:37.479762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:31:37.479762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:31:37.479762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:37.479762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:37.479762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:37.479762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:37.479762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:37.479762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:37.480956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:37.480956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:37.480956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:37.480956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:37.480956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:37.480956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:37.480956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:37.480956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:37.480956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:37.480956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:37.480956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:31:37.480956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:31:37.480956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 16 03:31:37.480956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:37.480956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 649/1298] 10.222.0.49 () {56 vars in 1117 bytes} [Fri Sep 16 03:31:37 2022] GET /identity/v3/services/placement => generated 90 bytes in 145 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:37.485593 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06e7273-8a7c-4301-ab49-3e19f5672fbe None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:37.504575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:37.582103 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06e7273-8a7c-4301-ab49-3e19f5672fbe None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:37.615589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435fb034-308a-4cc6-9244-48f0d0ba3395 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:37.616185 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435fb034-308a-4cc6-9244-48f0d0ba3395 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:37.616463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435fb034-308a-4cc6-9244-48f0d0ba3395 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:37.616732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435fb034-308a-4cc6-9244-48f0d0ba3395 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:37.617369 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435fb034-308a-4cc6-9244-48f0d0ba3395 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:37.617760 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435fb034-308a-4cc6-9244-48f0d0ba3395 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:37.619139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435fb034-308a-4cc6-9244-48f0d0ba3395 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:37.624243 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 650/1299] 10.222.0.49 () {56 vars in 1116 bytes} [Fri Sep 16 03:31:37 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:37.631199 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a3898c-360e-4b54-8cc7-2be6a1a4e8b2 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:37.662131 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:37.727962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a3898c-360e-4b54-8cc7-2be6a1a4e8b2 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:37.761630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fab3b65-6ff0-4b1f-bf6c-bd817f410229 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:37.762221 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fab3b65-6ff0-4b1f-bf6c-bd817f410229 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:37.762499 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fab3b65-6ff0-4b1f-bf6c-bd817f410229 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:37.762772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fab3b65-6ff0-4b1f-bf6c-bd817f410229 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:37.763494 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fab3b65-6ff0-4b1f-bf6c-bd817f410229 None admin] RBAC: Adding query filter params (interface=public, service_id=5157e75d40934af3b8a28a8fdd03bc47, region_id=RegionOne) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:37.763795 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fab3b65-6ff0-4b1f-bf6c-bd817f410229 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:37.765146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fab3b65-6ff0-4b1f-bf6c-bd817f410229 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:37.770062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 650/1300] 10.222.0.49 () {56 vars in 1251 bytes} [Fri Sep 16 03:31:37 2022] GET /identity/v3/endpoints?service_id=5157e75d40934af3b8a28a8fdd03bc47&interface=public®ion_id=RegionOne => generated 194 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:37.775838 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435fb034-308a-4cc6-9244-48f0d0ba3395 None admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:37.858958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:37.877160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435fb034-308a-4cc6-9244-48f0d0ba3395 None admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:37.910667 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464f54e8-2350-425c-bcba-0db2ce901004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:37.911291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464f54e8-2350-425c-bcba-0db2ce901004 None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:37.911569 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464f54e8-2350-425c-bcba-0db2ce901004 None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:37.911839 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464f54e8-2350-425c-bcba-0db2ce901004 None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:37.912427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464f54e8-2350-425c-bcba-0db2ce901004 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:37.913890 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464f54e8-2350-425c-bcba-0db2ce901004 None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:37.920461 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 651/1301] 10.222.0.49 () {56 vars in 1087 bytes} [Fri Sep 16 03:31:37 2022] GET /identity/v3/services => generated 2191 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:31:40.068926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4554aeb1-6374-4b27-a999-8e07e1d00805 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:40.069497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4554aeb1-6374-4b27-a999-8e07e1d00805 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:40.069497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4554aeb1-6374-4b27-a999-8e07e1d00805 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:40.070790 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 651/1302] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:31:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:31:40.075919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bb2bde-f3f5-4057-bcd9-a64dac1f4289 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:40.076324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bb2bde-f3f5-4057-bcd9-a64dac1f4289 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:40.076470 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bb2bde-f3f5-4057-bcd9-a64dac1f4289 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:40.094940 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-17bb2bde-f3f5-4057-bcd9-a64dac1f4289 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:31:40.096130 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-17bb2bde-f3f5-4057-bcd9-a64dac1f4289 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:31:40.162607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 652/1303] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:31:40 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 03:31:40.431097 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4554aeb1-6374-4b27-a999-8e07e1d00805 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:40.522967 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:40.529571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4554aeb1-6374-4b27-a999-8e07e1d00805 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:40.562188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61da928-a519-45c2-8162-53037bb78b01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:40.562810 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61da928-a519-45c2-8162-53037bb78b01 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:40.563121 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61da928-a519-45c2-8162-53037bb78b01 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:40.563403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61da928-a519-45c2-8162-53037bb78b01 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:40.563975 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61da928-a519-45c2-8162-53037bb78b01 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:40.565327 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61da928-a519-45c2-8162-53037bb78b01 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:40.574464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-b61da928-a519-45c2-8162-53037bb78b01 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 16 03:31:40.574464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:40.574464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 03:31:40.574464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 03:31:40.574464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 03:31:40.574464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 03:31:40.574464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 03:31:40.574464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:31:40.574464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 16 03:31:40.574464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:40.574464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 03:31:40.574464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:40.574464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 03:31:40.574464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 03:31:40.574464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 03:31:40.574464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 03:31:40.574464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 03:31:40.575676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 03:31:40.575676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 03:31:40.575676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 03:31:40.575676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 03:31:40.575676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 03:31:40.575676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 03:31:40.575676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 03:31:40.575676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 03:31:40.575676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 03:31:40.575676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 03:31:40.575676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 03:31:40.575676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 03:31:40.575676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 03:31:40.575676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 03:31:40.575676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 03:31:40.575676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 03:31:40.576573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 03:31:40.576573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Sep 16 03:31:40.576573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 03:31:40.576573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Sep 16 03:31:40.576573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 03:31:40.576573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 03:31:40.576573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 03:31:40.576573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Sep 16 03:31:40.576573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 03:31:40.576573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Sep 16 03:31:40.576573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 03:31:40.576573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 03:31:40.576573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 16 03:31:40.576573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Sep 16 03:31:40.576573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 652/1304] 10.222.0.49 () {56 vars in 1117 bytes} [Fri Sep 16 03:31:40 2022] GET /identity/v3/services/placement => generated 90 bytes in 145 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 03:31:40.581539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bb2bde-f3f5-4057-bcd9-a64dac1f4289 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:40.654705 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:40.677195 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bb2bde-f3f5-4057-bcd9-a64dac1f4289 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:40.714245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba0dddf-6e13-4db9-8bc8-4c1d36c83e9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:40.714848 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba0dddf-6e13-4db9-8bc8-4c1d36c83e9a None admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:40.715152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba0dddf-6e13-4db9-8bc8-4c1d36c83e9a None admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:40.715421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba0dddf-6e13-4db9-8bc8-4c1d36c83e9a None admin] PATH_INFO: `/v3/services` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:40.716041 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba0dddf-6e13-4db9-8bc8-4c1d36c83e9a None admin] RBAC: Adding query filter params (name=placement) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:31:40.716341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba0dddf-6e13-4db9-8bc8-4c1d36c83e9a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:40.717698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba0dddf-6e13-4db9-8bc8-4c1d36c83e9a None admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:40.722683 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 653/1305] 10.222.0.49 () {56 vars in 1116 bytes} [Fri Sep 16 03:31:40 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:31:40.728521 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61da928-a519-45c2-8162-53037bb78b01 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:31:40.827625 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61da928-a519-45c2-8162-53037bb78b01 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:31:40.860243 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa56de65-7b6a-473d-9187-ab08cbd9515f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:31:40.860843 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa56de65-7b6a-473d-9187-ab08cbd9515f None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:31:40.861124 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa56de65-7b6a-473d-9187-ab08cbd9515f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:31:40.861402 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa56de65-7b6a-473d-9187-ab08cbd9515f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:31:40.862056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa56de65-7b6a-473d-9187-ab08cbd9515f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:31:40.864344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa56de65-7b6a-473d-9187-ab08cbd9515f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:31:40.959623 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 653/1306] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:31:40 2022] POST /identity/v3/endpoints => generated 331 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:31:50.724313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=30 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:31:50.954352 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:26.429147 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c0e908-43d7-4a29-986e-91471036c134 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:26.429598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c0e908-43d7-4a29-986e-91471036c134 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:26.429680 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c0e908-43d7-4a29-986e-91471036c134 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:26.436169 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b652a3fe-f131-4767-8096-523f24910236 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:26.436570 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b652a3fe-f131-4767-8096-523f24910236 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:26.436711 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b652a3fe-f131-4767-8096-523f24910236 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:26.440016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 654/1307] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:39:26 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:39:26.528821 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-b652a3fe-f131-4767-8096-523f24910236 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:39:26.530043 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b652a3fe-f131-4767-8096-523f24910236 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:26.663548 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 654/1308] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:39:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:39:26.949603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a7548d-877f-42ea-be50-fa50e60a441f None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:26.950571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a7548d-877f-42ea-be50-fa50e60a441f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:26.951114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a7548d-877f-42ea-be50-fa50e60a441f None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:26.953096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 655/1309] 10.222.0.49 () {54 vars in 965 bytes} [Fri Sep 16 03:39:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:39:26.959706 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139d97d2-4a38-4417-8c1b-d5445fcf65cc None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:26.960314 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139d97d2-4a38-4417-8c1b-d5445fcf65cc None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:26.961058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139d97d2-4a38-4417-8c1b-d5445fcf65cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:27.056023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-139d97d2-4a38-4417-8c1b-d5445fcf65cc None None] MFA Rules not processed for user `f9275eeba471423e9fa2a31a6e08676d`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:39:27.057922 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-139d97d2-4a38-4417-8c1b-d5445fcf65cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:27.082227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:27.168466 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a7548d-877f-42ea-be50-fa50e60a441f None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:39:27.170981 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-98a7548d-877f-42ea-be50-fa50e60a441f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:27.175822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 655/1310] 10.222.0.49 () {58 vars in 1062 bytes} [Fri Sep 16 03:39:26 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:39:27.298383 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a7548d-877f-42ea-be50-fa50e60a441f None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:39:27.389301 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e00edc-75de-4305-99a8-e996b2c67cb0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:39:27.389915 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e00edc-75de-4305-99a8-e996b2c67cb0 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:27.390196 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e00edc-75de-4305-99a8-e996b2c67cb0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:27.390474 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e00edc-75de-4305-99a8-e996b2c67cb0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:27.392184 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e4e00edc-75de-4305-99a8-e996b2c67cb0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:27.453169 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:27.521984 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e00edc-75de-4305-99a8-e996b2c67cb0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:39:27.523792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e00edc-75de-4305-99a8-e996b2c67cb0 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:39:27.544492 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 656/1311] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:39:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:39:30.068335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1a2a6f-ecf0-49e3-a8ed-89670f7f9a65 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:30.068882 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1a2a6f-ecf0-49e3-a8ed-89670f7f9a65 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:30.068882 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1a2a6f-ecf0-49e3-a8ed-89670f7f9a65 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:30.070253 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 656/1312] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:39:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:39:30.076231 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86965e10-f2db-4234-809b-b6a2e14ce6d6 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:30.076795 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86965e10-f2db-4234-809b-b6a2e14ce6d6 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:30.076795 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86965e10-f2db-4234-809b-b6a2e14ce6d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:30.147717 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-86965e10-f2db-4234-809b-b6a2e14ce6d6 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:39:30.148930 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-86965e10-f2db-4234-809b-b6a2e14ce6d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:30.249436 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 657/1313] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:39:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:39:30.553101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35f558b-aa55-4b5d-a126-fa35ee95abc9 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:30.553384 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35f558b-aa55-4b5d-a126-fa35ee95abc9 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:30.553655 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35f558b-aa55-4b5d-a126-fa35ee95abc9 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:30.555012 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 657/1314] 10.222.0.49 () {54 vars in 965 bytes} [Fri Sep 16 03:39:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:39:30.560679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c82e8d-f070-4792-adc3-33edce9c740c None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:30.561053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c82e8d-f070-4792-adc3-33edce9c740c None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:30.561333 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c82e8d-f070-4792-adc3-33edce9c740c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:30.573562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:30.636566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-a6c82e8d-f070-4792-adc3-33edce9c740c None None] MFA Rules not processed for user `f9275eeba471423e9fa2a31a6e08676d`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:39:30.637771 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a6c82e8d-f070-4792-adc3-33edce9c740c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:30.711096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 658/1315] 10.222.0.49 () {58 vars in 1062 bytes} [Fri Sep 16 03:39:30 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:39:30.720116 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35f558b-aa55-4b5d-a126-fa35ee95abc9 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:39:30.721643 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b35f558b-aa55-4b5d-a126-fa35ee95abc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:30.823085 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35f558b-aa55-4b5d-a126-fa35ee95abc9 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:39:30.893744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:30.896084 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d95aa34-7270-46ea-b44c-134ffd804362 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:39:30.896687 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d95aa34-7270-46ea-b44c-134ffd804362 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:30.896969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d95aa34-7270-46ea-b44c-134ffd804362 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:30.897244 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d95aa34-7270-46ea-b44c-134ffd804362 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:30.898830 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9d95aa34-7270-46ea-b44c-134ffd804362 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:31.008108 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d95aa34-7270-46ea-b44c-134ffd804362 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:39:31.009864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d95aa34-7270-46ea-b44c-134ffd804362 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:39:31.028377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 658/1316] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:39:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:39:33.911416 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91ba2e2-3f9a-4d83-9806-fd6f4f8d3594 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:33.911961 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91ba2e2-3f9a-4d83-9806-fd6f4f8d3594 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:33.911961 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91ba2e2-3f9a-4d83-9806-fd6f4f8d3594 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:33.913308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 659/1317] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:39:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:39:33.918667 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2964cc-e37e-451a-978c-11e45bd16978 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:33.919084 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2964cc-e37e-451a-978c-11e45bd16978 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:33.919251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2964cc-e37e-451a-978c-11e45bd16978 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:33.992488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ac2964cc-e37e-451a-978c-11e45bd16978 None None] MFA Rules not processed for user `5eef2e8ac0a34e458d28dc3d163cb041`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:39:33.993683 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ac2964cc-e37e-451a-978c-11e45bd16978 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:34.044216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:34.087776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 659/1318] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:39:33 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:39:34.381162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91ba2e2-3f9a-4d83-9806-fd6f4f8d3594 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:39:34.382742 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c91ba2e2-3f9a-4d83-9806-fd6f4f8d3594 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:34.485355 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91ba2e2-3f9a-4d83-9806-fd6f4f8d3594 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:39:34.488703 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:34.561585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d02c22-5854-4d96-92e4-47fb657e8f4d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:39:34.562198 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d02c22-5854-4d96-92e4-47fb657e8f4d service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:34.562477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d02c22-5854-4d96-92e4-47fb657e8f4d service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:34.562749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d02c22-5854-4d96-92e4-47fb657e8f4d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:34.564362 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-60d02c22-5854-4d96-92e4-47fb657e8f4d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:34.671153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d02c22-5854-4d96-92e4-47fb657e8f4d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:39:34.672607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d02c22-5854-4d96-92e4-47fb657e8f4d service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:39:34.691284 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 660/1319] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:39:34 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:39:37.650515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f0737f-a2b0-4126-a6d5-f4b1908ca858 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:37.651056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f0737f-a2b0-4126-a6d5-f4b1908ca858 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:37.651056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f0737f-a2b0-4126-a6d5-f4b1908ca858 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:37.652378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 660/1320] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:39:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:39:37.657649 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd29481e-2032-4c53-a572-6486d4dafe85 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:37.658680 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd29481e-2032-4c53-a572-6486d4dafe85 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:37.658786 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd29481e-2032-4c53-a572-6486d4dafe85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:37.699886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:37.724751 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-cd29481e-2032-4c53-a572-6486d4dafe85 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:39:37.725967 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-cd29481e-2032-4c53-a572-6486d4dafe85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:37.821691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 661/1321] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:39:37 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:39:38.107617 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f0737f-a2b0-4126-a6d5-f4b1908ca858 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:39:38.109161 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-45f0737f-a2b0-4126-a6d5-f4b1908ca858 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:38.213140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f0737f-a2b0-4126-a6d5-f4b1908ca858 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:39:38.289172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988e0f97-0941-4e25-b4ec-db2e7be866f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:39:38.289775 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988e0f97-0941-4e25-b4ec-db2e7be866f8 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:38.290055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988e0f97-0941-4e25-b4ec-db2e7be866f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:38.290328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988e0f97-0941-4e25-b4ec-db2e7be866f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:38.292216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-988e0f97-0941-4e25-b4ec-db2e7be866f8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:38.300606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:38.408515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988e0f97-0941-4e25-b4ec-db2e7be866f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:39:38.410198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988e0f97-0941-4e25-b4ec-db2e7be866f8 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:39:38.428873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 661/1322] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:39:38 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:39:41.624477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5589ce4-79c6-47f0-941d-928e04eeda66 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:41.625044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5589ce4-79c6-47f0-941d-928e04eeda66 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:41.625044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5589ce4-79c6-47f0-941d-928e04eeda66 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:41.626286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 662/1323] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:39:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:39:41.631672 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1215629f-d060-4f7d-8209-20f4417ec50c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:41.632129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1215629f-d060-4f7d-8209-20f4417ec50c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:41.632364 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1215629f-d060-4f7d-8209-20f4417ec50c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:41.702110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1215629f-d060-4f7d-8209-20f4417ec50c None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:39:41.703390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1215629f-d060-4f7d-8209-20f4417ec50c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:41.765130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:41.809674 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 662/1324] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:39:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:39:42.096911 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5589ce4-79c6-47f0-941d-928e04eeda66 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:39:42.098415 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c5589ce4-79c6-47f0-941d-928e04eeda66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:42.201173 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5589ce4-79c6-47f0-941d-928e04eeda66 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:39:42.215758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:42.278455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886158ed-6ab1-4395-b350-163902931ab6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:39:42.279100 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886158ed-6ab1-4395-b350-163902931ab6 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:42.279382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886158ed-6ab1-4395-b350-163902931ab6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:42.279656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886158ed-6ab1-4395-b350-163902931ab6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:42.281241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-886158ed-6ab1-4395-b350-163902931ab6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:42.394605 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886158ed-6ab1-4395-b350-163902931ab6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:39:42.396086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886158ed-6ab1-4395-b350-163902931ab6 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:39:42.415290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 663/1325] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:39:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:39:44.649531 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d298c82-d29e-4b28-aa80-01801d2989f5 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:44.650095 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d298c82-d29e-4b28-aa80-01801d2989f5 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:44.650095 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d298c82-d29e-4b28-aa80-01801d2989f5 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:44.651619 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 663/1326] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:39:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:39:44.656672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a67f83e-bdd3-4069-8c1e-712e1b523df5 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:44.657062 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a67f83e-bdd3-4069-8c1e-712e1b523df5 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:44.657227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a67f83e-bdd3-4069-8c1e-712e1b523df5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:44.706559 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:44.724580 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-8a67f83e-bdd3-4069-8c1e-712e1b523df5 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:39:44.725786 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-8a67f83e-bdd3-4069-8c1e-712e1b523df5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:44.832768 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 664/1327] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:39:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:39:45.110447 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d298c82-d29e-4b28-aa80-01801d2989f5 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:39:45.113234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0d298c82-d29e-4b28-aa80-01801d2989f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:45.218290 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d298c82-d29e-4b28-aa80-01801d2989f5 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:39:45.295462 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33dd688a-9762-4e28-be43-84a937975b7f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:39:45.296073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dd688a-9762-4e28-be43-84a937975b7f service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:45.296355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dd688a-9762-4e28-be43-84a937975b7f service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:45.296632 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dd688a-9762-4e28-be43-84a937975b7f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:45.298277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-33dd688a-9762-4e28-be43-84a937975b7f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:45.319082 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:45.412842 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33dd688a-9762-4e28-be43-84a937975b7f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:39:45.414173 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33dd688a-9762-4e28-be43-84a937975b7f service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:39:45.432401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 664/1328] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:39:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:39:47.641328 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2eaf275-9ce1-4d41-a9cb-761590a15681 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:47.641883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2eaf275-9ce1-4d41-a9cb-761590a15681 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:47.641883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2eaf275-9ce1-4d41-a9cb-761590a15681 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:47.643805 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 665/1329] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:39:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:39:47.649220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8f15fd-0d14-4b83-98e4-635e5442e580 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:47.649676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8f15fd-0d14-4b83-98e4-635e5442e580 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:47.649785 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8f15fd-0d14-4b83-98e4-635e5442e580 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:47.718424 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-0d8f15fd-0d14-4b83-98e4-635e5442e580 None None] MFA Rules not processed for user `5eef2e8ac0a34e458d28dc3d163cb041`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:39:47.719672 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0d8f15fd-0d14-4b83-98e4-635e5442e580 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:47.787194 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:47.818426 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 665/1330] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:39:47 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:39:48.110001 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2eaf275-9ce1-4d41-a9cb-761590a15681 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:39:48.112081 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c2eaf275-9ce1-4d41-a9cb-761590a15681 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:48.236566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2eaf275-9ce1-4d41-a9cb-761590a15681 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:39:48.259997 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:48.321416 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1a5589-223b-45c2-bd68-3da059bc3f1d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:39:48.321947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1a5589-223b-45c2-bd68-3da059bc3f1d service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:48.325293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1a5589-223b-45c2-bd68-3da059bc3f1d service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:48.325293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1a5589-223b-45c2-bd68-3da059bc3f1d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:48.325293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-2d1a5589-223b-45c2-bd68-3da059bc3f1d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:48.444754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1a5589-223b-45c2-bd68-3da059bc3f1d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:39:48.446489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1a5589-223b-45c2-bd68-3da059bc3f1d service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:39:48.467232 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 666/1331] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:39:48 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:39:51.201514 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3ca0f3-7ca1-4127-84f8-f344e4ca721d None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:51.202057 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3ca0f3-7ca1-4127-84f8-f344e4ca721d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:51.202057 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3ca0f3-7ca1-4127-84f8-f344e4ca721d None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:51.203408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 666/1332] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:39:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:39:51.209264 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954fd8c0-abf2-433d-8a75-4ac47ba2454b None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:51.209733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954fd8c0-abf2-433d-8a75-4ac47ba2454b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:51.209840 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954fd8c0-abf2-433d-8a75-4ac47ba2454b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:51.273049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:51.275637 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-954fd8c0-abf2-433d-8a75-4ac47ba2454b None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:39:51.276829 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-954fd8c0-abf2-433d-8a75-4ac47ba2454b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:51.373642 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 667/1333] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:39:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:39:51.651145 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3ca0f3-7ca1-4127-84f8-f344e4ca721d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:39:51.652734 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4e3ca0f3-7ca1-4127-84f8-f344e4ca721d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:51.760856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3ca0f3-7ca1-4127-84f8-f344e4ca721d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:39:51.833336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff35d65-7a0e-4865-ab66-547b2912c6df service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:39:51.833939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff35d65-7a0e-4865-ab66-547b2912c6df service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:51.834218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff35d65-7a0e-4865-ab66-547b2912c6df service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:51.834492 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff35d65-7a0e-4865-ab66-547b2912c6df service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:51.836113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-eff35d65-7a0e-4865-ab66-547b2912c6df service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:51.863297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:51.944579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff35d65-7a0e-4865-ab66-547b2912c6df service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:39:51.946037 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff35d65-7a0e-4865-ab66-547b2912c6df service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:39:51.964966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 667/1334] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:39:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:39:54.171751 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb001d25-419d-4a08-9967-823c393f1c03 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:54.172321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb001d25-419d-4a08-9967-823c393f1c03 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:54.172321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb001d25-419d-4a08-9967-823c393f1c03 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:54.174254 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 668/1335] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:39:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:39:54.179557 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3c16ed-790d-4b92-8a77-01995b8e8185 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:54.179996 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3c16ed-790d-4b92-8a77-01995b8e8185 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:54.180101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3c16ed-790d-4b92-8a77-01995b8e8185 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:54.247708 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-fc3c16ed-790d-4b92-8a77-01995b8e8185 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:39:54.248907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fc3c16ed-790d-4b92-8a77-01995b8e8185 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:54.323796 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:54.347553 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 668/1336] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:39:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:39:54.649798 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb001d25-419d-4a08-9967-823c393f1c03 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:39:54.651348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-cb001d25-419d-4a08-9967-823c393f1c03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:54.755087 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb001d25-419d-4a08-9967-823c393f1c03 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:39:54.784674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:54.830720 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e174d89-9aa3-4eb4-9db2-bc7833ba2e92 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:39:54.831355 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e174d89-9aa3-4eb4-9db2-bc7833ba2e92 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:54.831639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e174d89-9aa3-4eb4-9db2-bc7833ba2e92 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:54.831911 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e174d89-9aa3-4eb4-9db2-bc7833ba2e92 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:54.833487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3e174d89-9aa3-4eb4-9db2-bc7833ba2e92 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:54.948412 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e174d89-9aa3-4eb4-9db2-bc7833ba2e92 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:39:54.949864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e174d89-9aa3-4eb4-9db2-bc7833ba2e92 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:39:54.971446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 669/1337] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:39:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:39:57.815263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af0d5d5-f3aa-40e9-aec7-5cf25b5799c7 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:57.815813 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af0d5d5-f3aa-40e9-aec7-5cf25b5799c7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:57.815813 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af0d5d5-f3aa-40e9-aec7-5cf25b5799c7 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:57.817158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 669/1338] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:39:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:39:57.822410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676d0ad6-7aa5-4428-a239-a2ce7b0b1b47 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:57.822831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676d0ad6-7aa5-4428-a239-a2ce7b0b1b47 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:57.822998 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676d0ad6-7aa5-4428-a239-a2ce7b0b1b47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:57.888062 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-676d0ad6-7aa5-4428-a239-a2ce7b0b1b47 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:39:57.889256 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-676d0ad6-7aa5-4428-a239-a2ce7b0b1b47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:57.898039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:57.984845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 670/1339] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:39:57 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:39:58.272566 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af0d5d5-f3aa-40e9-aec7-5cf25b5799c7 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:39:58.274132 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0af0d5d5-f3aa-40e9-aec7-5cf25b5799c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:58.383382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af0d5d5-f3aa-40e9-aec7-5cf25b5799c7 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:39:58.456570 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f078094-837e-444d-b5f7-6cdf7901e3d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:39:58.457188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f078094-837e-444d-b5f7-6cdf7901e3d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:39:58.457460 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f078094-837e-444d-b5f7-6cdf7901e3d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:39:58.457730 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f078094-837e-444d-b5f7-6cdf7901e3d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:39:58.459346 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9f078094-837e-444d-b5f7-6cdf7901e3d4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:39:58.495040 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:39:58.569391 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f078094-837e-444d-b5f7-6cdf7901e3d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:39:58.570834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f078094-837e-444d-b5f7-6cdf7901e3d4 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:39:58.588888 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 670/1340] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:39:58 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:40:02.915827 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970d6126-bf13-42d4-8c7a-7183ddf93ea1 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:02.916380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970d6126-bf13-42d4-8c7a-7183ddf93ea1 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:02.916482 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970d6126-bf13-42d4-8c7a-7183ddf93ea1 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:02.918326 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 671/1341] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:40:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:40:02.924547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4cfaa4-9fd7-4bc6-8377-0280e8c375c8 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:02.925013 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4cfaa4-9fd7-4bc6-8377-0280e8c375c8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:02.925131 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4cfaa4-9fd7-4bc6-8377-0280e8c375c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:02.993934 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-bf4cfaa4-9fd7-4bc6-8377-0280e8c375c8 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:02.995301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-bf4cfaa4-9fd7-4bc6-8377-0280e8c375c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:03.087836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:03.099017 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 671/1342] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:40:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:40:03.392763 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970d6126-bf13-42d4-8c7a-7183ddf93ea1 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:40:03.394305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-970d6126-bf13-42d4-8c7a-7183ddf93ea1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:03.501169 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970d6126-bf13-42d4-8c7a-7183ddf93ea1 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:40:03.540859 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:03.577746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69883786-357c-4d4d-b992-a5876757daf7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:40:03.578364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69883786-357c-4d4d-b992-a5876757daf7 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:03.578640 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69883786-357c-4d4d-b992-a5876757daf7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:03.578993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69883786-357c-4d4d-b992-a5876757daf7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:03.580577 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-69883786-357c-4d4d-b992-a5876757daf7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:03.697734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69883786-357c-4d4d-b992-a5876757daf7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:40:03.699207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69883786-357c-4d4d-b992-a5876757daf7 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:40:03.718321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 672/1343] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:40:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:40:06.871695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2867dacd-1a0c-4f4f-80b9-b9067e70fa67 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:06.872216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2867dacd-1a0c-4f4f-80b9-b9067e70fa67 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:06.872310 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2867dacd-1a0c-4f4f-80b9-b9067e70fa67 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:06.873716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 672/1344] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:40:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:40:06.879181 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12212438-b273-48bc-8da7-a713e124ca93 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:06.879596 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12212438-b273-48bc-8da7-a713e124ca93 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:06.879874 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12212438-b273-48bc-8da7-a713e124ca93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:06.951959 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-12212438-b273-48bc-8da7-a713e124ca93 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:06.953232 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-12212438-b273-48bc-8da7-a713e124ca93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:06.970878 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:07.049292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 673/1345] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:40:06 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:40:07.333362 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2867dacd-1a0c-4f4f-80b9-b9067e70fa67 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:40:07.334941 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2867dacd-1a0c-4f4f-80b9-b9067e70fa67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:07.444108 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2867dacd-1a0c-4f4f-80b9-b9067e70fa67 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:40:07.523480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb01fc33-e34c-4fff-93fd-db523571694d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:40:07.524089 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb01fc33-e34c-4fff-93fd-db523571694d service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:07.524369 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb01fc33-e34c-4fff-93fd-db523571694d service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:07.524645 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb01fc33-e34c-4fff-93fd-db523571694d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:07.526230 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-eb01fc33-e34c-4fff-93fd-db523571694d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:07.571088 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:07.661296 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb01fc33-e34c-4fff-93fd-db523571694d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:40:07.662964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb01fc33-e34c-4fff-93fd-db523571694d service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:40:07.690643 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 673/1346] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:40:07 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:40:12.201317 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8235ba46-b74e-4d7d-bd44-a243852ec8c3 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:12.201796 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8235ba46-b74e-4d7d-bd44-a243852ec8c3 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:12.201905 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8235ba46-b74e-4d7d-bd44-a243852ec8c3 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:12.203275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 674/1347] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:40:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:40:12.209732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d312d344-4851-47f4-956f-ee1ce77d9b87 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:12.210136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d312d344-4851-47f4-956f-ee1ce77d9b87 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:12.210136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d312d344-4851-47f4-956f-ee1ce77d9b87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:12.283561 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-d312d344-4851-47f4-956f-ee1ce77d9b87 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:12.285012 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d312d344-4851-47f4-956f-ee1ce77d9b87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:12.395161 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:12.397428 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 674/1348] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:40:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:40:12.693013 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8235ba46-b74e-4d7d-bd44-a243852ec8c3 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:40:12.694528 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-8235ba46-b74e-4d7d-bd44-a243852ec8c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:12.798602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8235ba46-b74e-4d7d-bd44-a243852ec8c3 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:40:12.847145 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:12.872567 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f987c5c8-313b-4a63-a79b-1efb2452667c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:40:12.873161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f987c5c8-313b-4a63-a79b-1efb2452667c service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:12.873445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f987c5c8-313b-4a63-a79b-1efb2452667c service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:12.873715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f987c5c8-313b-4a63-a79b-1efb2452667c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:12.875357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-f987c5c8-313b-4a63-a79b-1efb2452667c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:12.981854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f987c5c8-313b-4a63-a79b-1efb2452667c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:40:12.983324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f987c5c8-313b-4a63-a79b-1efb2452667c service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:40:13.002471 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 675/1349] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:40:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:40:17.936997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc1098c-bf67-4322-8b1b-2a25c200901e None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:17.937525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc1098c-bf67-4322-8b1b-2a25c200901e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:17.937525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc1098c-bf67-4322-8b1b-2a25c200901e None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:17.939511 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 675/1350] 10.222.0.49 () {54 vars in 921 bytes} [Fri Sep 16 03:40:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:40:17.944620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59f8958-dd44-46e9-bc16-99ac7785da7b None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:17.945036 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59f8958-dd44-46e9-bc16-99ac7785da7b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:17.945226 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59f8958-dd44-46e9-bc16-99ac7785da7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:18.085812 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-f59f8958-dd44-46e9-bc16-99ac7785da7b None None] MFA Rules not processed for user `176da4679fcb46148da9289fccdad5c2`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:18.090138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-f59f8958-dd44-46e9-bc16-99ac7785da7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:18.136054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:18.217699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 676/1351] 10.222.0.49 () {58 vars in 1018 bytes} [Fri Sep 16 03:40:17 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:40:22.431822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1738ba9-7dae-4755-a703-da2e57cebf4e None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:22.432379 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1738ba9-7dae-4755-a703-da2e57cebf4e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:22.432709 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1738ba9-7dae-4755-a703-da2e57cebf4e None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:22.440301 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0b99db-a284-4f6b-bb91-92260db73d7f None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:22.441170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0b99db-a284-4f6b-bb91-92260db73d7f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:22.441460 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0b99db-a284-4f6b-bb91-92260db73d7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:22.443250 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 676/1352] 10.222.0.49 () {54 vars in 921 bytes} [Fri Sep 16 03:40:22 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:40:22.544902 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-0d0b99db-a284-4f6b-bb91-92260db73d7f None None] MFA Rules not processed for user `176da4679fcb46148da9289fccdad5c2`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:22.547290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-0d0b99db-a284-4f6b-bb91-92260db73d7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:22.637048 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:22.681410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 677/1353] 10.222.0.49 () {58 vars in 1018 bytes} [Fri Sep 16 03:40:22 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:40:24.005094 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac129ad-e2c4-4df9-be3d-12e4c8547045 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:24.005969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac129ad-e2c4-4df9-be3d-12e4c8547045 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:24.006255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac129ad-e2c4-4df9-be3d-12e4c8547045 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:24.007630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 677/1354] 10.222.0.49 () {54 vars in 921 bytes} [Fri Sep 16 03:40:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:40:24.012714 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2e8892-b686-43ca-90e9-6ea602d1fd3f None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:24.013196 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2e8892-b686-43ca-90e9-6ea602d1fd3f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:24.013307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2e8892-b686-43ca-90e9-6ea602d1fd3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:24.091999 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-ab2e8892-b686-43ca-90e9-6ea602d1fd3f None None] MFA Rules not processed for user `176da4679fcb46148da9289fccdad5c2`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:24.093197 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ab2e8892-b686-43ca-90e9-6ea602d1fd3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:24.199715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 678/1355] 10.222.0.49 () {58 vars in 1018 bytes} [Fri Sep 16 03:40:24 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:40:27.940565 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb0ecd3-9b9f-4e54-832b-d380fd6d7e45 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:27.941088 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb0ecd3-9b9f-4e54-832b-d380fd6d7e45 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:27.941088 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb0ecd3-9b9f-4e54-832b-d380fd6d7e45 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:27.942536 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 678/1356] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:40:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:40:27.948645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43163881-8a8d-4ce8-8320-7c6a7a6c29d2 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:27.949077 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43163881-8a8d-4ce8-8320-7c6a7a6c29d2 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:27.949218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43163881-8a8d-4ce8-8320-7c6a7a6c29d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:27.995359 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:28.044519 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-43163881-8a8d-4ce8-8320-7c6a7a6c29d2 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:28.046642 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-43163881-8a8d-4ce8-8320-7c6a7a6c29d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:28.170670 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 679/1357] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:40:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:40:28.497230 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f098b3-3d48-452b-9c2d-798018531785 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:28.497861 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f098b3-3d48-452b-9c2d-798018531785 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:28.498397 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f098b3-3d48-452b-9c2d-798018531785 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:28.502265 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 679/1358] 10.222.0.49 () {54 vars in 962 bytes} [Fri Sep 16 03:40:28 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:40:28.512121 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b5b99d-1c47-4433-b46e-296fff68fac7 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:28.512417 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b5b99d-1c47-4433-b46e-296fff68fac7 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:28.512697 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b5b99d-1c47-4433-b46e-296fff68fac7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:28.650207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-d4b5b99d-1c47-4433-b46e-296fff68fac7 None None] MFA Rules not processed for user `ded1bcd10e62499a9aaa0ece5aa55c80`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:28.651657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d4b5b99d-1c47-4433-b46e-296fff68fac7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:28.708360 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:28.776903 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 680/1359] 10.222.0.49 () {58 vars in 1059 bytes} [Fri Sep 16 03:40:28 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:40:28.785534 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f098b3-3d48-452b-9c2d-798018531785 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:40:28.787077 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-29f098b3-3d48-452b-9c2d-798018531785 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:28.948229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f098b3-3d48-452b-9c2d-798018531785 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:40:29.034094 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:29.076946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb87ad9-ba44-4f8e-bd19-a48abfd14dd0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:40:29.077560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb87ad9-ba44-4f8e-bd19-a48abfd14dd0 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:29.077840 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb87ad9-ba44-4f8e-bd19-a48abfd14dd0 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:29.078120 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb87ad9-ba44-4f8e-bd19-a48abfd14dd0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:29.079773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7cb87ad9-ba44-4f8e-bd19-a48abfd14dd0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:29.175861 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb87ad9-ba44-4f8e-bd19-a48abfd14dd0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:40:29.177453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb87ad9-ba44-4f8e-bd19-a48abfd14dd0 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:40:29.196451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 680/1360] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:40:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:40:31.604202 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6222ad9c-dd12-469b-a911-bf36e67b2752 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:31.604715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6222ad9c-dd12-469b-a911-bf36e67b2752 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:31.604822 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6222ad9c-dd12-469b-a911-bf36e67b2752 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:31.606073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 681/1361] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:40:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:40:31.611931 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ba287f-9928-426f-8b9f-a9e5c14755c0 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:31.612405 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ba287f-9928-426f-8b9f-a9e5c14755c0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:31.612504 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ba287f-9928-426f-8b9f-a9e5c14755c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:31.680592 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f8ba287f-9928-426f-8b9f-a9e5c14755c0 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:31.681716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f8ba287f-9928-426f-8b9f-a9e5c14755c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:31.720762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:31.782650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 681/1362] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:40:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:40:32.072269 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6222ad9c-dd12-469b-a911-bf36e67b2752 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:40:32.073875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-6222ad9c-dd12-469b-a911-bf36e67b2752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:32.270482 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6222ad9c-dd12-469b-a911-bf36e67b2752 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:40:32.280202 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:32.386956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feda6449-56d4-489b-ac0b-51da1f16b038 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:40:32.387567 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feda6449-56d4-489b-ac0b-51da1f16b038 service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:32.387845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feda6449-56d4-489b-ac0b-51da1f16b038 service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:32.388110 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feda6449-56d4-489b-ac0b-51da1f16b038 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:32.389678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-feda6449-56d4-489b-ac0b-51da1f16b038 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:32.482112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feda6449-56d4-489b-ac0b-51da1f16b038 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:40:32.483814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feda6449-56d4-489b-ac0b-51da1f16b038 service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:40:32.502752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 682/1363] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:40:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 431 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:40:35.090965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b67c898-f87e-4804-8002-4b5bebdf5180 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:35.091554 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b67c898-f87e-4804-8002-4b5bebdf5180 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:35.091554 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b67c898-f87e-4804-8002-4b5bebdf5180 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:35.093088 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 682/1364] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:40:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:40:35.099013 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caef4a74-ea26-4f2c-ba7b-acb259278527 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:35.099476 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caef4a74-ea26-4f2c-ba7b-acb259278527 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:35.099601 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caef4a74-ea26-4f2c-ba7b-acb259278527 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:35.116710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:35.169505 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-caef4a74-ea26-4f2c-ba7b-acb259278527 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:35.170684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-caef4a74-ea26-4f2c-ba7b-acb259278527 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:35.277979 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 683/1365] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:40:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:40:35.568553 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115e4610-6562-40d8-a072-69ddaa5a7616 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:35.568888 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115e4610-6562-40d8-a072-69ddaa5a7616 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:35.569108 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115e4610-6562-40d8-a072-69ddaa5a7616 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:35.570639 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 683/1366] 10.222.0.49 () {54 vars in 962 bytes} [Fri Sep 16 03:40:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:40:35.575967 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9295f14-44d2-42d7-854a-4ca7c498b5d1 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:35.576254 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9295f14-44d2-42d7-854a-4ca7c498b5d1 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:35.576816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9295f14-44d2-42d7-854a-4ca7c498b5d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:35.661580 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-b9295f14-44d2-42d7-854a-4ca7c498b5d1 None None] MFA Rules not processed for user `ded1bcd10e62499a9aaa0ece5aa55c80`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:35.662810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b9295f14-44d2-42d7-854a-4ca7c498b5d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:35.701756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:35.790697 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 684/1367] 10.222.0.49 () {58 vars in 1059 bytes} [Fri Sep 16 03:40:35 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:40:35.799209 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115e4610-6562-40d8-a072-69ddaa5a7616 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:40:35.800716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-115e4610-6562-40d8-a072-69ddaa5a7616 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:35.955093 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115e4610-6562-40d8-a072-69ddaa5a7616 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:40:35.976939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:36.083069 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09ee9bd-2771-45e6-95ca-503d5afa6713 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:40:36.083678 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09ee9bd-2771-45e6-95ca-503d5afa6713 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:36.083961 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09ee9bd-2771-45e6-95ca-503d5afa6713 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:36.084237 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09ee9bd-2771-45e6-95ca-503d5afa6713 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:36.085823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e09ee9bd-2771-45e6-95ca-503d5afa6713 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:36.124123 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d7f418-047d-4fa5-acde-5cedff49aef9 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:36.124658 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d7f418-047d-4fa5-acde-5cedff49aef9 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:36.124904 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d7f418-047d-4fa5-acde-5cedff49aef9 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:36.131101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 685/1368] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:40:36 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:40:36.132416 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57c1618-a5c0-4e07-ba2e-dfe2c85c95ae None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:36.132772 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57c1618-a5c0-4e07-ba2e-dfe2c85c95ae None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:36.133115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57c1618-a5c0-4e07-ba2e-dfe2c85c95ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:36.183024 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09ee9bd-2771-45e6-95ca-503d5afa6713 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:40:36.184659 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09ee9bd-2771-45e6-95ca-503d5afa6713 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:40:36.206458 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 684/1369] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:40:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:40:36.208563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-e57c1618-a5c0-4e07-ba2e-dfe2c85c95ae None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:36.209792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e57c1618-a5c0-4e07-ba2e-dfe2c85c95ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:36.306513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:36.314715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 686/1370] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:40:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:40:36.624463 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7609ca-fb21-4dd5-9679-0c80146628f7 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:36.624851 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7609ca-fb21-4dd5-9679-0c80146628f7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:36.625199 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7609ca-fb21-4dd5-9679-0c80146628f7 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:36.626948 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 685/1371] 10.222.0.49 () {54 vars in 964 bytes} [Fri Sep 16 03:40:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:40:36.632004 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce53c5f9-b826-4e25-859c-0269262e5fbc None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:36.632368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce53c5f9-b826-4e25-859c-0269262e5fbc None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:36.632717 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce53c5f9-b826-4e25-859c-0269262e5fbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:36.727482 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-ce53c5f9-b826-4e25-859c-0269262e5fbc None None] MFA Rules not processed for user `e6aeffd924734771bc5bf0d282c9052d`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:36.728594 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ce53c5f9-b826-4e25-859c-0269262e5fbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:36.816889 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 687/1372] 10.222.0.49 () {58 vars in 1061 bytes} [Fri Sep 16 03:40:36 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:40:36.824592 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7609ca-fb21-4dd5-9679-0c80146628f7 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:40:36.826110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7c7609ca-fb21-4dd5-9679-0c80146628f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:36.856894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:36.943064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7609ca-fb21-4dd5-9679-0c80146628f7 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:40:37.030636 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1591409-27f0-471b-9c4f-6ca0a94e318b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:40:37.031420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1591409-27f0-471b-9c4f-6ca0a94e318b service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:37.031704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1591409-27f0-471b-9c4f-6ca0a94e318b service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:37.032579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1591409-27f0-471b-9c4f-6ca0a94e318b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:37.034284 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b1591409-27f0-471b-9c4f-6ca0a94e318b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:37.118840 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:37.155885 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1591409-27f0-471b-9c4f-6ca0a94e318b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:40:37.157924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1591409-27f0-471b-9c4f-6ca0a94e318b service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:40:37.178676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 686/1373] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:40:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:40:38.601300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348ab93e-7cba-4c78-a554-e5b60d20c139 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:38.601824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348ab93e-7cba-4c78-a554-e5b60d20c139 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:38.601924 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348ab93e-7cba-4c78-a554-e5b60d20c139 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:38.603319 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 688/1374] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:40:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:40:38.608780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e500d533-4947-44ba-94e8-6f654f8affec None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:38.609192 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e500d533-4947-44ba-94e8-6f654f8affec None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:38.609352 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e500d533-4947-44ba-94e8-6f654f8affec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:38.679736 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e500d533-4947-44ba-94e8-6f654f8affec None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:38.680946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e500d533-4947-44ba-94e8-6f654f8affec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:38.778870 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 687/1375] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:40:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:40:39.074063 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348ab93e-7cba-4c78-a554-e5b60d20c139 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:40:39.075660 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-348ab93e-7cba-4c78-a554-e5b60d20c139 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:39.173916 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:39.244917 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348ab93e-7cba-4c78-a554-e5b60d20c139 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:40:39.373029 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1e0db0-0f29-4436-8a1b-13b4ea244fcf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:40:39.373645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1e0db0-0f29-4436-8a1b-13b4ea244fcf service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:39.373925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1e0db0-0f29-4436-8a1b-13b4ea244fcf service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:39.374197 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1e0db0-0f29-4436-8a1b-13b4ea244fcf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:39.375945 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-8a1e0db0-0f29-4436-8a1b-13b4ea244fcf service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:39.428788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:39.472846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1e0db0-0f29-4436-8a1b-13b4ea244fcf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:40:39.475605 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1e0db0-0f29-4436-8a1b-13b4ea244fcf service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:40:39.496385 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 689/1376] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:40:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:40:41.936840 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d5c38b-553d-4c41-86c5-2fd299418a4e None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:41.937417 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d5c38b-553d-4c41-86c5-2fd299418a4e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:41.937417 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d5c38b-553d-4c41-86c5-2fd299418a4e None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:41.939690 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 688/1377] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:40:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:40:41.944826 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1611e244-db49-42ed-b815-ee9e84fe0b76 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:41.945269 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1611e244-db49-42ed-b815-ee9e84fe0b76 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:41.945362 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1611e244-db49-42ed-b815-ee9e84fe0b76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:42.015154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-1611e244-db49-42ed-b815-ee9e84fe0b76 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:42.017229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1611e244-db49-42ed-b815-ee9e84fe0b76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:42.127357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 690/1378] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:40:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:40:42.419668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d5c38b-553d-4c41-86c5-2fd299418a4e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:40:42.421356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-71d5c38b-553d-4c41-86c5-2fd299418a4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:42.462542 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:42.580642 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d5c38b-553d-4c41-86c5-2fd299418a4e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:40:42.700085 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:42.705785 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef26f1e6-3c5a-416c-8184-fcb7c2e85274 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:40:42.706611 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef26f1e6-3c5a-416c-8184-fcb7c2e85274 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:42.706840 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef26f1e6-3c5a-416c-8184-fcb7c2e85274 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:42.707157 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef26f1e6-3c5a-416c-8184-fcb7c2e85274 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:42.708759 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ef26f1e6-3c5a-416c-8184-fcb7c2e85274 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:42.804555 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef26f1e6-3c5a-416c-8184-fcb7c2e85274 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:40:42.806160 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef26f1e6-3c5a-416c-8184-fcb7c2e85274 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:40:42.825461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 689/1379] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:40:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:40:45.426083 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ddd7ce-f070-4806-9e5d-e336ac6d3854 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:45.426746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ddd7ce-f070-4806-9e5d-e336ac6d3854 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:45.426746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ddd7ce-f070-4806-9e5d-e336ac6d3854 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:45.428384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 691/1380] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:40:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:40:45.433787 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a213dc5d-7183-4c06-91f4-33333aff4f4d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:45.434264 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a213dc5d-7183-4c06-91f4-33333aff4f4d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:45.434375 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a213dc5d-7183-4c06-91f4-33333aff4f4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:45.515347 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a213dc5d-7183-4c06-91f4-33333aff4f4d None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:45.516910 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a213dc5d-7183-4c06-91f4-33333aff4f4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:45.592273 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:45.632397 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 690/1381] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:40:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:40:45.996718 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ddd7ce-f070-4806-9e5d-e336ac6d3854 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:40:45.998533 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a0ddd7ce-f070-4806-9e5d-e336ac6d3854 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:46.030496 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:46.173888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ddd7ce-f070-4806-9e5d-e336ac6d3854 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:40:46.280414 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:46.298004 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f72762a-190b-4a4c-983d-5de6a8683649 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:40:46.298620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f72762a-190b-4a4c-983d-5de6a8683649 service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:46.301978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f72762a-190b-4a4c-983d-5de6a8683649 service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:46.301978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f72762a-190b-4a4c-983d-5de6a8683649 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:46.301978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-8f72762a-190b-4a4c-983d-5de6a8683649 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:46.410472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f72762a-190b-4a4c-983d-5de6a8683649 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:40:46.412315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f72762a-190b-4a4c-983d-5de6a8683649 service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:40:46.433610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 692/1382] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:40:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 437 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:40:49.336782 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45223fb-911e-4498-b0bd-b4adb2ddfbd1 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:49.337324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45223fb-911e-4498-b0bd-b4adb2ddfbd1 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:49.337324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45223fb-911e-4498-b0bd-b4adb2ddfbd1 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:49.338703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 691/1383] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:40:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:40:49.344464 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a9cfec-4280-471e-b73c-dc529e399523 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:49.345256 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a9cfec-4280-471e-b73c-dc529e399523 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:49.345256 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a9cfec-4280-471e-b73c-dc529e399523 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:49.418957 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-b1a9cfec-4280-471e-b73c-dc529e399523 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:49.420169 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b1a9cfec-4280-471e-b73c-dc529e399523 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:49.483778 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:49.530165 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 693/1384] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:40:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:40:49.834926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45223fb-911e-4498-b0bd-b4adb2ddfbd1 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:40:49.836749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a45223fb-911e-4498-b0bd-b4adb2ddfbd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:49.996808 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45223fb-911e-4498-b0bd-b4adb2ddfbd1 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:40:50.060746 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:50.125277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3832a614-1756-4517-af69-ff71e988ec86 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:40:50.127301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3832a614-1756-4517-af69-ff71e988ec86 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:50.127648 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3832a614-1756-4517-af69-ff71e988ec86 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:50.127983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3832a614-1756-4517-af69-ff71e988ec86 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:50.129775 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3832a614-1756-4517-af69-ff71e988ec86 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:50.232718 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3832a614-1756-4517-af69-ff71e988ec86 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:40:50.234328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3832a614-1756-4517-af69-ff71e988ec86 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:40:50.257177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 692/1385] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:40:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:40:52.582691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8e395e-8bf7-47c9-9974-2f942f7384a3 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:52.583249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8e395e-8bf7-47c9-9974-2f942f7384a3 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:52.583249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8e395e-8bf7-47c9-9974-2f942f7384a3 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:52.584674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 694/1386] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:40:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:40:52.590129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc42268-f77a-44dc-88c2-b93dc49a7357 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:52.590553 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc42268-f77a-44dc-88c2-b93dc49a7357 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:52.590677 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc42268-f77a-44dc-88c2-b93dc49a7357 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:52.658053 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-fdc42268-f77a-44dc-88c2-b93dc49a7357 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:52.659319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fdc42268-f77a-44dc-88c2-b93dc49a7357 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:52.668398 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:52.760211 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 693/1387] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:40:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:40:53.046565 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8e395e-8bf7-47c9-9974-2f942f7384a3 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:40:53.048855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-6d8e395e-8bf7-47c9-9974-2f942f7384a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:53.203992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8e395e-8bf7-47c9-9974-2f942f7384a3 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:40:53.253635 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:53.333665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc34879a-3def-491c-8c96-a6f8331feea9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:40:53.334510 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc34879a-3def-491c-8c96-a6f8331feea9 service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:53.334775 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc34879a-3def-491c-8c96-a6f8331feea9 service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:53.335084 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc34879a-3def-491c-8c96-a6f8331feea9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:53.336683 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-cc34879a-3def-491c-8c96-a6f8331feea9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:53.435807 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc34879a-3def-491c-8c96-a6f8331feea9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:40:53.437397 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc34879a-3def-491c-8c96-a6f8331feea9 service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:40:53.458339 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 695/1388] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:40:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:40:53.660381 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a86687f-f3ea-414f-9490-d9f38998410b None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:53.660902 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a86687f-f3ea-414f-9490-d9f38998410b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:53.660902 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a86687f-f3ea-414f-9490-d9f38998410b None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:53.662250 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 694/1389] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:40:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:40:53.667557 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2f954b-99e7-4b95-8452-b2e601f61c7d None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:53.668421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2f954b-99e7-4b95-8452-b2e601f61c7d None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:53.668421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2f954b-99e7-4b95-8452-b2e601f61c7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:53.741403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:53.744699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-6a2f954b-99e7-4b95-8452-b2e601f61c7d None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:53.745955 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-6a2f954b-99e7-4b95-8452-b2e601f61c7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:53.856384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 696/1390] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:40:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:40:54.165362 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d4fa11-0b2c-423b-9542-1d7df9a092a9 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:54.165696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d4fa11-0b2c-423b-9542-1d7df9a092a9 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:54.166018 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d4fa11-0b2c-423b-9542-1d7df9a092a9 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:54.167603 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 695/1391] 10.222.0.49 () {54 vars in 962 bytes} [Fri Sep 16 03:40:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:40:54.174194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2ba905-2c1f-4174-bc83-f17ab880b1c4 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:54.174486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2ba905-2c1f-4174-bc83-f17ab880b1c4 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:54.174761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2ba905-2c1f-4174-bc83-f17ab880b1c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:54.264906 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-9c2ba905-2c1f-4174-bc83-f17ab880b1c4 None None] MFA Rules not processed for user `05ceb9fc31c345ccabaee8f30a3982a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:54.266163 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-9c2ba905-2c1f-4174-bc83-f17ab880b1c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:54.347543 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:54.355604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 697/1392] 10.222.0.49 () {58 vars in 1059 bytes} [Fri Sep 16 03:40:54 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:40:54.365636 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d4fa11-0b2c-423b-9542-1d7df9a092a9 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:40:54.367192 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a5d4fa11-0b2c-423b-9542-1d7df9a092a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:54.483877 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d4fa11-0b2c-423b-9542-1d7df9a092a9 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:40:54.524402 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:54.561366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9401902c-4be5-46fd-8f14-17922cc8cfe2 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ceb9fc31c345ccabaee8f30a3982a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:40:54.561988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9401902c-4be5-46fd-8f14-17922cc8cfe2 service glance] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:54.562269 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9401902c-4be5-46fd-8f14-17922cc8cfe2 service glance] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:54.562540 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9401902c-4be5-46fd-8f14-17922cc8cfe2 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:54.564840 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9401902c-4be5-46fd-8f14-17922cc8cfe2 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:54.692509 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9401902c-4be5-46fd-8f14-17922cc8cfe2 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:40:54.694206 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9401902c-4be5-46fd-8f14-17922cc8cfe2 service glance] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:40:54.718506 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 696/1393] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:40:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:40:55.663421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15f6c87-ca9a-4f66-858c-b3232593e95e None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:55.663950 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15f6c87-ca9a-4f66-858c-b3232593e95e None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:55.664067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15f6c87-ca9a-4f66-858c-b3232593e95e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:55.733005 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-e15f6c87-ca9a-4f66-858c-b3232593e95e None None] MFA Rules not processed for user `d40810f61ead4c3da414194b325da80a`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:55.734209 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e15f6c87-ca9a-4f66-858c-b3232593e95e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:55.830270 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 698/1394] 10.222.0.49 () {58 vars in 1009 bytes} [Fri Sep 16 03:40:55 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:40:55.878935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e1d57a-ef99-4a40-9b43-24722bbe21e2 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:55.879456 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e1d57a-ef99-4a40-9b43-24722bbe21e2 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:55.879650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e1d57a-ef99-4a40-9b43-24722bbe21e2 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:55.880947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 697/1395] 10.222.0.49 () {54 vars in 943 bytes} [Fri Sep 16 03:40:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:40:55.887878 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422aeafd-6edd-4f39-bec8-c8d489bc6fed None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:55.888222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422aeafd-6edd-4f39-bec8-c8d489bc6fed None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:55.888554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422aeafd-6edd-4f39-bec8-c8d489bc6fed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:55.894092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25bd016-361c-42b1-a32b-4fbdac25b4e8 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:55.894383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25bd016-361c-42b1-a32b-4fbdac25b4e8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:55.894659 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25bd016-361c-42b1-a32b-4fbdac25b4e8 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:55.896005 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 698/1396] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:40:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:40:55.901180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83283ca-d5ea-44bd-8fe6-0eaa88fa5f20 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:55.901463 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83283ca-d5ea-44bd-8fe6-0eaa88fa5f20 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:55.901741 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83283ca-d5ea-44bd-8fe6-0eaa88fa5f20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:55.970928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-d83283ca-d5ea-44bd-8fe6-0eaa88fa5f20 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:55.972165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d83283ca-d5ea-44bd-8fe6-0eaa88fa5f20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:55.974747 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-422aeafd-6edd-4f39-bec8-c8d489bc6fed None None] MFA Rules not processed for user `aa30da9972e140d7905c522576681429`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:55.975959 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-422aeafd-6edd-4f39-bec8-c8d489bc6fed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:55.992565 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:56.004134 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:56.078786 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 699/1397] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:40:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:40:56.088979 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 699/1398] 10.222.0.49 () {58 vars in 1040 bytes} [Fri Sep 16 03:40:55 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:40:56.099254 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83283ca-d5ea-44bd-8fe6-0eaa88fa5f20 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:40:56.100472 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d83283ca-d5ea-44bd-8fe6-0eaa88fa5f20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:56.203436 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83283ca-d5ea-44bd-8fe6-0eaa88fa5f20 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:40:56.258699 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:56.316343 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfa7e62-66be-40dc-a81b-6dba8c2ba5ea service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa30da9972e140d7905c522576681429', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:40:56.316967 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfa7e62-66be-40dc-a81b-6dba8c2ba5ea service swift] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:56.317245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfa7e62-66be-40dc-a81b-6dba8c2ba5ea service swift] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:56.317517 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfa7e62-66be-40dc-a81b-6dba8c2ba5ea service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:56.319198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9bfa7e62-66be-40dc-a81b-6dba8c2ba5ea service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:56.381442 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422aeafd-6edd-4f39-bec8-c8d489bc6fed None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:40:56.382731 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-422aeafd-6edd-4f39-bec8-c8d489bc6fed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:56.410088 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfa7e62-66be-40dc-a81b-6dba8c2ba5ea service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:40:56.411736 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfa7e62-66be-40dc-a81b-6dba8c2ba5ea service swift] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:40:56.420012 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 700/1399] 10.222.0.49 () {60 vars in 1384 bytes} [Fri Sep 16 03:40:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 03:40:56.484781 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422aeafd-6edd-4f39-bec8-c8d489bc6fed None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:40:56.534936 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:56.598116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a41bd8e-2576-4612-8320-9c40823eeaaf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:40:56.598724 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a41bd8e-2576-4612-8320-9c40823eeaaf service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:56.599038 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a41bd8e-2576-4612-8320-9c40823eeaaf service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:56.599314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a41bd8e-2576-4612-8320-9c40823eeaaf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:56.600898 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-6a41bd8e-2576-4612-8320-9c40823eeaaf service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:56.695320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a41bd8e-2576-4612-8320-9c40823eeaaf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:40:56.696786 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a41bd8e-2576-4612-8320-9c40823eeaaf service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:40:56.716539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 700/1400] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:40:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:40:59.161528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bd3d4e-7914-4a6a-8aab-5f620c101715 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:59.162207 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bd3d4e-7914-4a6a-8aab-5f620c101715 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:59.162514 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bd3d4e-7914-4a6a-8aab-5f620c101715 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:59.166959 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 701/1401] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:40:59 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:40:59.170259 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3319ef89-34a3-4aba-b72a-26e79c05a81f None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:59.170860 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3319ef89-34a3-4aba-b72a-26e79c05a81f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:59.170860 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3319ef89-34a3-4aba-b72a-26e79c05a81f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:59.248001 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-3319ef89-34a3-4aba-b72a-26e79c05a81f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:40:59.249218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3319ef89-34a3-4aba-b72a-26e79c05a81f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:59.252433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:59.364465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 701/1402] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:40:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:40:59.664572 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bd3d4e-7914-4a6a-8aab-5f620c101715 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:40:59.667034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e9bd3d4e-7914-4a6a-8aab-5f620c101715 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:40:59.770122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:40:59.840911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bd3d4e-7914-4a6a-8aab-5f620c101715 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:40:59.970029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19de3037-a63f-4ed0-911a-416face6c8be service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:40:59.970648 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19de3037-a63f-4ed0-911a-416face6c8be service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:40:59.970966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19de3037-a63f-4ed0-911a-416face6c8be service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:40:59.971237 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19de3037-a63f-4ed0-911a-416face6c8be service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:40:59.972905 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-19de3037-a63f-4ed0-911a-416face6c8be service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:00.021067 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:00.072928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19de3037-a63f-4ed0-911a-416face6c8be service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:41:00.074561 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19de3037-a63f-4ed0-911a-416face6c8be service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:41:00.094007 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 702/1403] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:40:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:41:02.472748 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46694d5-c32d-4292-b547-5b9fe61f909a None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:02.473770 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46694d5-c32d-4292-b547-5b9fe61f909a None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:02.474052 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46694d5-c32d-4292-b547-5b9fe61f909a None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:02.475469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 702/1404] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:41:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:41:02.481078 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ebe3dd-d469-49d4-a95e-ae1d6529ff4c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:02.481592 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ebe3dd-d469-49d4-a95e-ae1d6529ff4c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:02.481686 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ebe3dd-d469-49d4-a95e-ae1d6529ff4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:02.558494 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-89ebe3dd-d469-49d4-a95e-ae1d6529ff4c None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:41:02.560234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-89ebe3dd-d469-49d4-a95e-ae1d6529ff4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:02.681543 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 703/1405] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:41:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:41:02.973773 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46694d5-c32d-4292-b547-5b9fe61f909a None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:41:02.975321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a46694d5-c32d-4292-b547-5b9fe61f909a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:03.125603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:03.141829 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46694d5-c32d-4292-b547-5b9fe61f909a None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:41:03.261660 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a509437-faf7-45dc-8af7-4def48f7afc8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:41:03.262193 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a509437-faf7-45dc-8af7-4def48f7afc8 service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:03.262638 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a509437-faf7-45dc-8af7-4def48f7afc8 service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:03.262944 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a509437-faf7-45dc-8af7-4def48f7afc8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:03.264532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-0a509437-faf7-45dc-8af7-4def48f7afc8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:03.369353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a509437-faf7-45dc-8af7-4def48f7afc8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:41:03.370982 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a509437-faf7-45dc-8af7-4def48f7afc8 service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:41:03.381280 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:03.390974 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 703/1406] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:41:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:41:05.942311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed449586-5784-461e-b916-553c17f2d175 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:05.942858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed449586-5784-461e-b916-553c17f2d175 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:05.942858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed449586-5784-461e-b916-553c17f2d175 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:05.944247 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 704/1407] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:41:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:41:05.949524 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bbf6dd-1420-4506-986c-04f881f20e27 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:05.949964 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bbf6dd-1420-4506-986c-04f881f20e27 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:05.950064 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bbf6dd-1420-4506-986c-04f881f20e27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:06.024819 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-04bbf6dd-1420-4506-986c-04f881f20e27 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:41:06.026145 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-04bbf6dd-1420-4506-986c-04f881f20e27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:06.131675 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 704/1408] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:41:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:41:06.426947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed449586-5784-461e-b916-553c17f2d175 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:41:06.428472 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ed449586-5784-461e-b916-553c17f2d175 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:06.452139 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:06.603242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed449586-5784-461e-b916-553c17f2d175 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:41:06.708339 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:06.734425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27cca67-ebfd-4706-a5f7-d32e3302b11e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:41:06.735130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27cca67-ebfd-4706-a5f7-d32e3302b11e service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:06.735414 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27cca67-ebfd-4706-a5f7-d32e3302b11e service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:06.735691 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27cca67-ebfd-4706-a5f7-d32e3302b11e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:06.737506 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a27cca67-ebfd-4706-a5f7-d32e3302b11e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:06.837328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27cca67-ebfd-4706-a5f7-d32e3302b11e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:41:06.839198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27cca67-ebfd-4706-a5f7-d32e3302b11e service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:41:06.859206 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 705/1409] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:41:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:41:16.129918 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=68 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:16.856383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=54 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:20.783804 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bea5bb-bcd5-4580-b28d-93078a5539c5 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:20.784361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bea5bb-bcd5-4580-b28d-93078a5539c5 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:20.784361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bea5bb-bcd5-4580-b28d-93078a5539c5 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:20.785705 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 705/1410] 10.222.0.49 () {54 vars in 918 bytes} [Fri Sep 16 03:41:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:41:20.791028 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4e4ce2-9632-4df5-8f8c-0860c011caae None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:20.791435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4e4ce2-9632-4df5-8f8c-0860c011caae None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:20.791591 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4e4ce2-9632-4df5-8f8c-0860c011caae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:20.867405 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-6b4e4ce2-9632-4df5-8f8c-0860c011caae None None] MFA Rules not processed for user `176da4679fcb46148da9289fccdad5c2`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:41:20.868612 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6b4e4ce2-9632-4df5-8f8c-0860c011caae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:20.974301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 706/1411] 10.222.0.49 () {58 vars in 1015 bytes} [Fri Sep 16 03:41:20 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:41:20.989725 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc4ae3f-9e8f-4d20-b4fe-ac1978a4142e None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:20.990017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc4ae3f-9e8f-4d20-b4fe-ac1978a4142e None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:20.990298 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc4ae3f-9e8f-4d20-b4fe-ac1978a4142e None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:20.991717 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 706/1412] 10.222.0.49 () {54 vars in 918 bytes} [Fri Sep 16 03:41:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:41:20.996611 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5a2033-05b5-4609-8693-e93433cef3df None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:20.996902 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5a2033-05b5-4609-8693-e93433cef3df None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:20.997184 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5a2033-05b5-4609-8693-e93433cef3df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:21.071307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:21.073752 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-eb5a2033-05b5-4609-8693-e93433cef3df None None] MFA Rules not processed for user `ded1bcd10e62499a9aaa0ece5aa55c80`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:41:21.074976 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-eb5a2033-05b5-4609-8693-e93433cef3df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:21.189474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 707/1413] 10.222.0.49 () {58 vars in 1015 bytes} [Fri Sep 16 03:41:20 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:41:23.962984 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814dcf00-126f-4c97-8178-ee3380014f0f None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:23.963519 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814dcf00-126f-4c97-8178-ee3380014f0f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:23.963519 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814dcf00-126f-4c97-8178-ee3380014f0f None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:23.964803 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 707/1414] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 03:41:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:41:23.970612 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7d0df5-774a-4772-9387-a4743c5e783e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:23.971100 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7d0df5-774a-4772-9387-a4743c5e783e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:23.971212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7d0df5-774a-4772-9387-a4743c5e783e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:24.037895 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ec7d0df5-774a-4772-9387-a4743c5e783e None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:41:24.039131 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ec7d0df5-774a-4772-9387-a4743c5e783e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:24.092978 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:24.134675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 708/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 03:41:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:41:24.424580 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814dcf00-126f-4c97-8178-ee3380014f0f None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:41:24.426075 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-814dcf00-126f-4c97-8178-ee3380014f0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:24.571907 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814dcf00-126f-4c97-8178-ee3380014f0f None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:41:24.643610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:24.683962 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df902dae-4b6e-4757-b42d-ca6f6146b517 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:41:24.684568 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df902dae-4b6e-4757-b42d-ca6f6146b517 service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:24.684860 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df902dae-4b6e-4757-b42d-ca6f6146b517 service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:24.685137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df902dae-4b6e-4757-b42d-ca6f6146b517 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:24.686707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-df902dae-4b6e-4757-b42d-ca6f6146b517 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:24.775787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df902dae-4b6e-4757-b42d-ca6f6146b517 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:41:24.780015 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df902dae-4b6e-4757-b42d-ca6f6146b517 service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:41:24.795644 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 708/1416] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:41:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:41:25.944529 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ff492d-04fe-47ac-8d81-c4bee393f3a9 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:25.945105 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ff492d-04fe-47ac-8d81-c4bee393f3a9 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:25.945105 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ff492d-04fe-47ac-8d81-c4bee393f3a9 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:25.946417 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 709/1417] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 03:41:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:41:25.952857 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f041d1-5ca0-423f-9b9d-e168a338bb1b None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:25.953803 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f041d1-5ca0-423f-9b9d-e168a338bb1b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:25.954142 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f041d1-5ca0-423f-9b9d-e168a338bb1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:26.031582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-85f041d1-5ca0-423f-9b9d-e168a338bb1b None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:41:26.032773 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-85f041d1-5ca0-423f-9b9d-e168a338bb1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:26.059102 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:26.127578 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 709/1418] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 03:41:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:41:26.279398 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ff492d-04fe-47ac-8d81-c4bee393f3a9 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:41:26.280946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f5ff492d-04fe-47ac-8d81-c4bee393f3a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:26.436738 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ff492d-04fe-47ac-8d81-c4bee393f3a9 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:41:26.591141 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:26.663601 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038974f6-b322-4f23-81bb-a00a4df77afc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:41:26.664208 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038974f6-b322-4f23-81bb-a00a4df77afc service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:26.664482 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038974f6-b322-4f23-81bb-a00a4df77afc service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:26.664747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038974f6-b322-4f23-81bb-a00a4df77afc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:26.666362 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-038974f6-b322-4f23-81bb-a00a4df77afc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:26.755603 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038974f6-b322-4f23-81bb-a00a4df77afc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:41:26.757196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038974f6-b322-4f23-81bb-a00a4df77afc service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:41:26.774736 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 710/1419] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:41:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 496 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:41:27.978617 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57177da7-e4c4-41b6-a06a-3ffab09ebae4 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:27.979228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57177da7-e4c4-41b6-a06a-3ffab09ebae4 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:27.979228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57177da7-e4c4-41b6-a06a-3ffab09ebae4 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:27.980547 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 710/1420] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 03:41:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:41:27.987677 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e81d84-ef85-43ed-bb96-d51521008d67 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:27.988125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e81d84-ef85-43ed-bb96-d51521008d67 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:27.988222 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e81d84-ef85-43ed-bb96-d51521008d67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:28.041679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:28.064470 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-26e81d84-ef85-43ed-bb96-d51521008d67 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:41:28.065679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-26e81d84-ef85-43ed-bb96-d51521008d67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:28.169117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 711/1421] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 03:41:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:41:28.322822 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57177da7-e4c4-41b6-a06a-3ffab09ebae4 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:41:28.324398 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-57177da7-e4c4-41b6-a06a-3ffab09ebae4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:28.472599 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57177da7-e4c4-41b6-a06a-3ffab09ebae4 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:41:28.486334 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:28.589267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27887fc-bd25-4de4-9f48-aa0a8f1aada9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:41:28.589804 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27887fc-bd25-4de4-9f48-aa0a8f1aada9 service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:28.590088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27887fc-bd25-4de4-9f48-aa0a8f1aada9 service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:28.590364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27887fc-bd25-4de4-9f48-aa0a8f1aada9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:28.592031 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d27887fc-bd25-4de4-9f48-aa0a8f1aada9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:28.684132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27887fc-bd25-4de4-9f48-aa0a8f1aada9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:41:28.685742 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27887fc-bd25-4de4-9f48-aa0a8f1aada9 service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:41:28.704298 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 711/1422] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:41:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:41:29.833170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c38eb7-ffa0-4141-9b5e-d45dfccec613 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:29.833728 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c38eb7-ffa0-4141-9b5e-d45dfccec613 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:29.833728 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c38eb7-ffa0-4141-9b5e-d45dfccec613 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:29.835074 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 712/1423] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 03:41:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:41:29.842525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6d2d65-510b-48d7-9562-b0506ccc422b None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:29.843005 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6d2d65-510b-48d7-9562-b0506ccc422b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:29.843117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6d2d65-510b-48d7-9562-b0506ccc422b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:29.864168 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:29.919719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-1a6d2d65-510b-48d7-9562-b0506ccc422b None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:41:29.920899 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1a6d2d65-510b-48d7-9562-b0506ccc422b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:30.014866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 712/1424] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 03:41:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:41:30.169348 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c38eb7-ffa0-4141-9b5e-d45dfccec613 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:41:30.170780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e4c38eb7-ffa0-4141-9b5e-d45dfccec613 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:30.292878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:30.330813 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c38eb7-ffa0-4141-9b5e-d45dfccec613 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:41:30.443376 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194324dd-99d5-4bbf-af8b-3e9bc0ca94ca service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:41:30.443986 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194324dd-99d5-4bbf-af8b-3e9bc0ca94ca service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:30.444269 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194324dd-99d5-4bbf-af8b-3e9bc0ca94ca service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:30.444545 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194324dd-99d5-4bbf-af8b-3e9bc0ca94ca service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:30.446138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-194324dd-99d5-4bbf-af8b-3e9bc0ca94ca service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:30.527124 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:30.541544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194324dd-99d5-4bbf-af8b-3e9bc0ca94ca service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:41:30.548280 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194324dd-99d5-4bbf-af8b-3e9bc0ca94ca service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:41:30.561720 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 713/1425] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:41:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:41:31.676834 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99d1d95-a2ab-4ae5-a078-0e4c8792374c None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:31.677418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99d1d95-a2ab-4ae5-a078-0e4c8792374c None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:31.677418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99d1d95-a2ab-4ae5-a078-0e4c8792374c None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:31.678762 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 713/1426] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 03:41:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:41:31.684916 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dd9619-751a-4611-a93e-9a7a00d7c37c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:31.685397 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dd9619-751a-4611-a93e-9a7a00d7c37c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:31.685505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dd9619-751a-4611-a93e-9a7a00d7c37c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:31.759526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e9dd9619-751a-4611-a93e-9a7a00d7c37c None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:41:31.760723 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e9dd9619-751a-4611-a93e-9a7a00d7c37c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:31.854901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 714/1427] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 03:41:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:41:32.007817 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99d1d95-a2ab-4ae5-a078-0e4c8792374c None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:41:32.009340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e99d1d95-a2ab-4ae5-a078-0e4c8792374c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:32.092640 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:32.152690 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99d1d95-a2ab-4ae5-a078-0e4c8792374c None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:41:32.262479 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3504c1c4-1b98-40bc-be03-f86d329f6297 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:41:32.263073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3504c1c4-1b98-40bc-be03-f86d329f6297 service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:32.263352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3504c1c4-1b98-40bc-be03-f86d329f6297 service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:32.263623 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3504c1c4-1b98-40bc-be03-f86d329f6297 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:32.265180 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3504c1c4-1b98-40bc-be03-f86d329f6297 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:32.314802 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:32.355515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3504c1c4-1b98-40bc-be03-f86d329f6297 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:41:32.357151 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3504c1c4-1b98-40bc-be03-f86d329f6297 service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:41:32.375433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 714/1428] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:41:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:41:33.504594 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0471aff-224f-46d9-96f7-559dab7c5f04 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:33.505165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0471aff-224f-46d9-96f7-559dab7c5f04 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:33.505165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0471aff-224f-46d9-96f7-559dab7c5f04 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:33.506450 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 715/1429] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 03:41:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:41:33.512857 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82beb5ef-9bbf-4d17-b81f-d87b5e3edbcd None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:33.513321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82beb5ef-9bbf-4d17-b81f-d87b5e3edbcd None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:33.513434 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82beb5ef-9bbf-4d17-b81f-d87b5e3edbcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:33.588329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-82beb5ef-9bbf-4d17-b81f-d87b5e3edbcd None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:41:33.589433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-82beb5ef-9bbf-4d17-b81f-d87b5e3edbcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:33.682698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 715/1430] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 03:41:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:41:33.834215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0471aff-224f-46d9-96f7-559dab7c5f04 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:41:33.835746 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a0471aff-224f-46d9-96f7-559dab7c5f04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:33.884732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:33.978845 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0471aff-224f-46d9-96f7-559dab7c5f04 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:41:34.096604 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59d123b-181e-4eb2-97bf-9b0d7e17f938 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:41:34.097208 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59d123b-181e-4eb2-97bf-9b0d7e17f938 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:34.097485 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59d123b-181e-4eb2-97bf-9b0d7e17f938 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:34.097755 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59d123b-181e-4eb2-97bf-9b0d7e17f938 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:34.099371 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e59d123b-181e-4eb2-97bf-9b0d7e17f938 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:34.106141 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:34.189860 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59d123b-181e-4eb2-97bf-9b0d7e17f938 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:41:34.191396 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59d123b-181e-4eb2-97bf-9b0d7e17f938 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:41:34.208711 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 716/1431] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:41:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:41:35.336214 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bacd34-e8f7-43fa-9903-00fcd4f2367b None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:35.336730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bacd34-e8f7-43fa-9903-00fcd4f2367b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:35.336836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bacd34-e8f7-43fa-9903-00fcd4f2367b None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:35.338123 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 716/1432] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 03:41:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:41:35.344906 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f10875-3894-43e9-8c72-9011531f2a67 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:35.345356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f10875-3894-43e9-8c72-9011531f2a67 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:35.345459 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f10875-3894-43e9-8c72-9011531f2a67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:35.420698 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-87f10875-3894-43e9-8c72-9011531f2a67 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:41:35.421916 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-87f10875-3894-43e9-8c72-9011531f2a67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:35.487585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:35.516491 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 717/1433] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 03:41:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:41:35.664809 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bacd34-e8f7-43fa-9903-00fcd4f2367b None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:41:35.666216 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-89bacd34-e8f7-43fa-9903-00fcd4f2367b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:35.683906 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:35.811011 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bacd34-e8f7-43fa-9903-00fcd4f2367b None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:41:35.898126 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:35.923449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf798a40-397d-402f-92f7-f0af6749e6b6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:41:35.923995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf798a40-397d-402f-92f7-f0af6749e6b6 service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:35.924280 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf798a40-397d-402f-92f7-f0af6749e6b6 service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:35.924561 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf798a40-397d-402f-92f7-f0af6749e6b6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:35.926140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-cf798a40-397d-402f-92f7-f0af6749e6b6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:36.017508 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf798a40-397d-402f-92f7-f0af6749e6b6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:41:36.019167 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf798a40-397d-402f-92f7-f0af6749e6b6 service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:41:36.037661 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 717/1434] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:41:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:41:37.205349 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11b3082-ab22-4091-9993-5006ae314fb1 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:37.205940 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11b3082-ab22-4091-9993-5006ae314fb1 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:37.205940 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11b3082-ab22-4091-9993-5006ae314fb1 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:37.207259 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 718/1435] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 03:41:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:41:37.213560 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3789f882-b997-4cdd-8c6d-be44a919103b None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:37.213972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3789f882-b997-4cdd-8c6d-be44a919103b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:37.214135 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3789f882-b997-4cdd-8c6d-be44a919103b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:37.293169 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-3789f882-b997-4cdd-8c6d-be44a919103b None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:41:37.294661 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3789f882-b997-4cdd-8c6d-be44a919103b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:37.337545 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:37.393628 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 718/1436] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 03:41:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:41:37.543474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11b3082-ab22-4091-9993-5006ae314fb1 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:41:37.545157 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a11b3082-ab22-4091-9993-5006ae314fb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:37.696513 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11b3082-ab22-4091-9993-5006ae314fb1 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:41:37.749195 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:37.807472 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c4d797-02a6-462b-93b4-de3a24926026 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:41:37.808008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c4d797-02a6-462b-93b4-de3a24926026 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:37.808289 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c4d797-02a6-462b-93b4-de3a24926026 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:37.808563 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c4d797-02a6-462b-93b4-de3a24926026 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:37.810153 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-08c4d797-02a6-462b-93b4-de3a24926026 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:37.898194 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c4d797-02a6-462b-93b4-de3a24926026 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:41:37.899821 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c4d797-02a6-462b-93b4-de3a24926026 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:41:37.917723 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 719/1437] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:41:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:41:39.029242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a352f6-10e5-4cef-bfec-a67295f2f628 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:39.029788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a352f6-10e5-4cef-bfec-a67295f2f628 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:39.029788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a352f6-10e5-4cef-bfec-a67295f2f628 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:39.031146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 719/1438] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 03:41:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:41:39.037581 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6135f5-3b43-4dc4-97d4-baeb12c4c243 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:39.038018 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6135f5-3b43-4dc4-97d4-baeb12c4c243 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:39.038129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6135f5-3b43-4dc4-97d4-baeb12c4c243 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:39.114316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-4e6135f5-3b43-4dc4-97d4-baeb12c4c243 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:41:39.115559 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4e6135f5-3b43-4dc4-97d4-baeb12c4c243 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:39.117583 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:39.215550 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 720/1439] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 03:41:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:41:39.365939 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a352f6-10e5-4cef-bfec-a67295f2f628 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:41:39.367462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-85a352f6-10e5-4cef-bfec-a67295f2f628 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:39.517321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a352f6-10e5-4cef-bfec-a67295f2f628 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:41:39.544897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:39.630957 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae600acc-f984-42f0-931c-e5e47042bec7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:41:39.631490 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae600acc-f984-42f0-931c-e5e47042bec7 service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:39.631771 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae600acc-f984-42f0-931c-e5e47042bec7 service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:39.632045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae600acc-f984-42f0-931c-e5e47042bec7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:39.633636 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ae600acc-f984-42f0-931c-e5e47042bec7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:39.724263 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae600acc-f984-42f0-931c-e5e47042bec7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:41:39.726441 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae600acc-f984-42f0-931c-e5e47042bec7 service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:41:39.744442 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 720/1440] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:41:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:41:41.870184 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b76b5b3-ef00-410a-bab9-30954aa5fa1e None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:41.870769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b76b5b3-ef00-410a-bab9-30954aa5fa1e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:41.870769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b76b5b3-ef00-410a-bab9-30954aa5fa1e None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:41.872088 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 721/1441] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 03:41:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:41:41.877545 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939cf9ab-1443-450f-aad3-06b6253ff6fa None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:41.877948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939cf9ab-1443-450f-aad3-06b6253ff6fa None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:41.878094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939cf9ab-1443-450f-aad3-06b6253ff6fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:41.918514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:41.943956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-939cf9ab-1443-450f-aad3-06b6253ff6fa None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:41:41.945139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-939cf9ab-1443-450f-aad3-06b6253ff6fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:42.042530 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 721/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 03:41:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:41:42.327495 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b76b5b3-ef00-410a-bab9-30954aa5fa1e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:41:42.329068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2b76b5b3-ef00-410a-bab9-30954aa5fa1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:42.431704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b76b5b3-ef00-410a-bab9-30954aa5fa1e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:41:42.464831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:42.506690 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6d41ee-64a1-4691-998e-64b21d48e8f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:41:42.507321 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6d41ee-64a1-4691-998e-64b21d48e8f8 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:42.507603 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6d41ee-64a1-4691-998e-64b21d48e8f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:42.507877 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6d41ee-64a1-4691-998e-64b21d48e8f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:42.509450 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6a6d41ee-64a1-4691-998e-64b21d48e8f8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:42.628621 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6d41ee-64a1-4691-998e-64b21d48e8f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:41:42.629975 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6d41ee-64a1-4691-998e-64b21d48e8f8 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:41:42.648172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 722/1443] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:41:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:41:45.029235 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4b6872-fec5-4dd5-a8cd-8613a3198923 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:45.029857 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4b6872-fec5-4dd5-a8cd-8613a3198923 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:45.029857 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4b6872-fec5-4dd5-a8cd-8613a3198923 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:45.031173 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 722/1444] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 03:41:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:41:45.036923 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb7fda3-ce1e-47c1-be29-261675d82746 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:45.037371 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb7fda3-ce1e-47c1-be29-261675d82746 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:45.037481 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb7fda3-ce1e-47c1-be29-261675d82746 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:45.103778 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-5bb7fda3-ce1e-47c1-be29-261675d82746 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:41:45.104974 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5bb7fda3-ce1e-47c1-be29-261675d82746 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:45.117996 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:45.204041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 723/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 03:41:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:41:45.491433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4b6872-fec5-4dd5-a8cd-8613a3198923 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:41:45.492970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1b4b6872-fec5-4dd5-a8cd-8613a3198923 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:45.596703 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4b6872-fec5-4dd5-a8cd-8613a3198923 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:41:45.609721 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:45.670516 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790fdac6-f665-4581-a8cb-6ede45ccd053 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:41:45.671149 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790fdac6-f665-4581-a8cb-6ede45ccd053 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:45.671434 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790fdac6-f665-4581-a8cb-6ede45ccd053 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:45.671709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790fdac6-f665-4581-a8cb-6ede45ccd053 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:45.673399 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-790fdac6-f665-4581-a8cb-6ede45ccd053 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:45.781759 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790fdac6-f665-4581-a8cb-6ede45ccd053 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:41:45.783372 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790fdac6-f665-4581-a8cb-6ede45ccd053 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:41:45.801187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 723/1446] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:41:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:41:48.329145 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601243cb-b48f-49dc-875f-32453cd43f2f None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:48.329572 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601243cb-b48f-49dc-875f-32453cd43f2f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:48.329715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601243cb-b48f-49dc-875f-32453cd43f2f None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:48.331080 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 724/1447] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 03:41:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:41:48.336307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34169274-b939-45e9-b201-f492b96a526b None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:48.336711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34169274-b939-45e9-b201-f492b96a526b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:48.336861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34169274-b939-45e9-b201-f492b96a526b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:48.384326 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:48.402990 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-34169274-b939-45e9-b201-f492b96a526b None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:41:48.408836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-34169274-b939-45e9-b201-f492b96a526b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:48.503435 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 724/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 03:41:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:41:48.787465 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601243cb-b48f-49dc-875f-32453cd43f2f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:41:48.789001 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-601243cb-b48f-49dc-875f-32453cd43f2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:48.895806 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601243cb-b48f-49dc-875f-32453cd43f2f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:41:48.935584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:48.968266 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfec6a27-a82f-4b2e-a9c1-b9d3cb5f8ec5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:41:48.968922 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfec6a27-a82f-4b2e-a9c1-b9d3cb5f8ec5 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:48.969267 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfec6a27-a82f-4b2e-a9c1-b9d3cb5f8ec5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:48.969541 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfec6a27-a82f-4b2e-a9c1-b9d3cb5f8ec5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:48.971196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-dfec6a27-a82f-4b2e-a9c1-b9d3cb5f8ec5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:49.080551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfec6a27-a82f-4b2e-a9c1-b9d3cb5f8ec5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:41:49.081963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfec6a27-a82f-4b2e-a9c1-b9d3cb5f8ec5 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:41:49.099981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 725/1449] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:41:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:41:51.382829 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7789a6b5-3bf1-4084-80b0-f5b36246073e None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:51.383381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7789a6b5-3bf1-4084-80b0-f5b36246073e None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:51.383381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7789a6b5-3bf1-4084-80b0-f5b36246073e None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:51.384738 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 725/1450] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 03:41:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:41:51.389879 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e05a7c0-bb2d-4eb3-a6c6-93ab54342747 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:51.390279 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e05a7c0-bb2d-4eb3-a6c6-93ab54342747 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:51.390421 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e05a7c0-bb2d-4eb3-a6c6-93ab54342747 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:51.456091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1e05a7c0-bb2d-4eb3-a6c6-93ab54342747 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:41:51.457293 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1e05a7c0-bb2d-4eb3-a6c6-93ab54342747 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:51.477460 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:51.556904 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 726/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 03:41:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:41:51.838632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7789a6b5-3bf1-4084-80b0-f5b36246073e None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:41:51.840259 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-7789a6b5-3bf1-4084-80b0-f5b36246073e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:51.944284 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7789a6b5-3bf1-4084-80b0-f5b36246073e None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:41:51.965293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:52.019043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78628964-1808-42f4-8484-e4f5d9b16160 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:41:52.019648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78628964-1808-42f4-8484-e4f5d9b16160 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:52.019924 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78628964-1808-42f4-8484-e4f5d9b16160 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:52.020191 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78628964-1808-42f4-8484-e4f5d9b16160 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:52.021749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-78628964-1808-42f4-8484-e4f5d9b16160 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:52.129072 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78628964-1808-42f4-8484-e4f5d9b16160 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:41:52.130439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78628964-1808-42f4-8484-e4f5d9b16160 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:41:52.148693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 726/1452] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:41:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:41:54.443526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f20749-b7e2-44cd-baa8-5cdf0669d926 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:54.444113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f20749-b7e2-44cd-baa8-5cdf0669d926 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:54.444113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f20749-b7e2-44cd-baa8-5cdf0669d926 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:54.445413 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 727/1453] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 03:41:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:41:54.450673 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5f0871-5020-465a-9814-721d151b1675 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:54.451103 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5f0871-5020-465a-9814-721d151b1675 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:54.451249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5f0871-5020-465a-9814-721d151b1675 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:54.514270 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:54.517529 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-eb5f0871-5020-465a-9814-721d151b1675 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:41:54.518660 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-eb5f0871-5020-465a-9814-721d151b1675 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:54.614587 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 727/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 03:41:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:41:54.901639 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f20749-b7e2-44cd-baa8-5cdf0669d926 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:41:54.903229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d3f20749-b7e2-44cd-baa8-5cdf0669d926 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:55.006557 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f20749-b7e2-44cd-baa8-5cdf0669d926 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:41:55.056126 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:55.080085 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cab3a86-0bdb-4e85-be4c-fe16e9fb0022 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:41:55.080679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cab3a86-0bdb-4e85-be4c-fe16e9fb0022 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:55.080956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cab3a86-0bdb-4e85-be4c-fe16e9fb0022 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:55.081218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cab3a86-0bdb-4e85-be4c-fe16e9fb0022 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:55.082772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0cab3a86-0bdb-4e85-be4c-fe16e9fb0022 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:55.194053 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cab3a86-0bdb-4e85-be4c-fe16e9fb0022 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:41:55.195528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cab3a86-0bdb-4e85-be4c-fe16e9fb0022 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:41:55.213391 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 728/1455] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:41:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:41:57.499441 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97d1a30-2ee0-4ceb-9d55-f10d1148e296 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:57.500069 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97d1a30-2ee0-4ceb-9d55-f10d1148e296 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:57.500069 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97d1a30-2ee0-4ceb-9d55-f10d1148e296 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:57.501406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 728/1456] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 03:41:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:41:57.507176 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d39f057-b161-4877-91e2-afdc390d60c2 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:57.507690 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d39f057-b161-4877-91e2-afdc390d60c2 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:57.507874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d39f057-b161-4877-91e2-afdc390d60c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:57.574801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-4d39f057-b161-4877-91e2-afdc390d60c2 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:41:57.576058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4d39f057-b161-4877-91e2-afdc390d60c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:57.613086 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:57.682372 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 729/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 03:41:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:41:57.972910 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97d1a30-2ee0-4ceb-9d55-f10d1148e296 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:41:57.974422 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d97d1a30-2ee0-4ceb-9d55-f10d1148e296 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:58.076500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97d1a30-2ee0-4ceb-9d55-f10d1148e296 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:41:58.105728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:41:58.149163 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63dd38ba-95ad-49d1-8333-c5ecdceb2c14 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:41:58.149761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dd38ba-95ad-49d1-8333-c5ecdceb2c14 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:41:58.150042 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dd38ba-95ad-49d1-8333-c5ecdceb2c14 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:41:58.150312 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dd38ba-95ad-49d1-8333-c5ecdceb2c14 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:41:58.151926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-63dd38ba-95ad-49d1-8333-c5ecdceb2c14 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:41:58.261212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63dd38ba-95ad-49d1-8333-c5ecdceb2c14 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:41:58.262588 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63dd38ba-95ad-49d1-8333-c5ecdceb2c14 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:41:58.281643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 729/1458] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:41:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:42:00.532167 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d608d4-4194-4eff-bd5a-8af631f8d02a None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:00.532720 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d608d4-4194-4eff-bd5a-8af631f8d02a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:00.532720 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d608d4-4194-4eff-bd5a-8af631f8d02a None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:00.533977 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 730/1459] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 03:42:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:42:00.539276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8193bae-1a29-4be9-aebb-88e87c042069 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:00.539730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8193bae-1a29-4be9-aebb-88e87c042069 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:00.539846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8193bae-1a29-4be9-aebb-88e87c042069 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:00.605190 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-e8193bae-1a29-4be9-aebb-88e87c042069 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:42:00.606420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e8193bae-1a29-4be9-aebb-88e87c042069 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:00.615410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:00.703500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 730/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 03:42:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:42:00.986572 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d608d4-4194-4eff-bd5a-8af631f8d02a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:42:00.988142 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-94d608d4-4194-4eff-bd5a-8af631f8d02a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:01.092354 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d608d4-4194-4eff-bd5a-8af631f8d02a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:42:01.147583 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:01.166167 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdac5343-d6b8-43b9-bf9b-6849480a7e40 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:42:01.166761 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdac5343-d6b8-43b9-bf9b-6849480a7e40 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:01.167065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdac5343-d6b8-43b9-bf9b-6849480a7e40 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:01.167338 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdac5343-d6b8-43b9-bf9b-6849480a7e40 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:01.168897 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cdac5343-d6b8-43b9-bf9b-6849480a7e40 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:01.276041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdac5343-d6b8-43b9-bf9b-6849480a7e40 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:42:01.277406 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdac5343-d6b8-43b9-bf9b-6849480a7e40 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:42:01.296064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 731/1461] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:42:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:42:03.496398 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ec27ae-7b8f-4347-b1be-6be26cde463d None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:03.496965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ec27ae-7b8f-4347-b1be-6be26cde463d None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:03.496965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ec27ae-7b8f-4347-b1be-6be26cde463d None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:03.498286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 731/1462] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 03:42:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:42:03.503681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3a7356-23ee-445d-b62a-6218f75b61d7 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:03.504112 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3a7356-23ee-445d-b62a-6218f75b61d7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:03.504217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3a7356-23ee-445d-b62a-6218f75b61d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:03.569765 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7f3a7356-23ee-445d-b62a-6218f75b61d7 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:42:03.570982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7f3a7356-23ee-445d-b62a-6218f75b61d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:03.608190 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:03.666583 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 732/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 03:42:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:42:03.969963 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ec27ae-7b8f-4347-b1be-6be26cde463d None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:42:03.971511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-67ec27ae-7b8f-4347-b1be-6be26cde463d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:04.081311 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ec27ae-7b8f-4347-b1be-6be26cde463d None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:42:04.117958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:04.153176 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064906d3-fdb7-4498-a63b-5e96f417f9af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:42:04.153806 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064906d3-fdb7-4498-a63b-5e96f417f9af service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:04.154083 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064906d3-fdb7-4498-a63b-5e96f417f9af service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:04.154351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064906d3-fdb7-4498-a63b-5e96f417f9af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:04.155944 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-064906d3-fdb7-4498-a63b-5e96f417f9af service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:04.264479 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064906d3-fdb7-4498-a63b-5e96f417f9af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:42:04.265923 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064906d3-fdb7-4498-a63b-5e96f417f9af service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:42:04.285527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 732/1464] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:42:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:42:06.453858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7541a578-4608-4e97-89e7-e92b5d0f0eff None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:06.454416 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7541a578-4608-4e97-89e7-e92b5d0f0eff None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:06.454416 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7541a578-4608-4e97-89e7-e92b5d0f0eff None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:06.455764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 733/1465] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 03:42:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:42:06.460990 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85d76de-c723-4006-ab33-64ab60a05a57 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:06.461408 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85d76de-c723-4006-ab33-64ab60a05a57 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:06.461544 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85d76de-c723-4006-ab33-64ab60a05a57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:06.527595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-c85d76de-c723-4006-ab33-64ab60a05a57 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:42:06.528711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c85d76de-c723-4006-ab33-64ab60a05a57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:06.548085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:06.626519 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 733/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 03:42:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:42:06.941231 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7541a578-4608-4e97-89e7-e92b5d0f0eff None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:42:06.942815 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7541a578-4608-4e97-89e7-e92b5d0f0eff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:07.048170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7541a578-4608-4e97-89e7-e92b5d0f0eff None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:42:07.113439 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:07.119831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4561afd6-be49-40b4-b68e-9b1f74dd2292 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:42:07.120432 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4561afd6-be49-40b4-b68e-9b1f74dd2292 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:07.120711 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4561afd6-be49-40b4-b68e-9b1f74dd2292 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:07.120985 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4561afd6-be49-40b4-b68e-9b1f74dd2292 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:07.122575 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4561afd6-be49-40b4-b68e-9b1f74dd2292 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:07.231708 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4561afd6-be49-40b4-b68e-9b1f74dd2292 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:42:07.233140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4561afd6-be49-40b4-b68e-9b1f74dd2292 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:42:07.251875 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 734/1467] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:42:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:42:09.429177 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2fa74e-b1bf-4068-ab5c-5cea1c89e3ba None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:09.429743 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2fa74e-b1bf-4068-ab5c-5cea1c89e3ba None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:09.429743 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2fa74e-b1bf-4068-ab5c-5cea1c89e3ba None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:09.431117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 734/1468] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 03:42:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:42:09.436314 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d0c329-67b2-4229-b493-0b2415d41d29 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:09.436772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d0c329-67b2-4229-b493-0b2415d41d29 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:09.436890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d0c329-67b2-4229-b493-0b2415d41d29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:09.502081 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a0d0c329-67b2-4229-b493-0b2415d41d29 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:42:09.503308 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a0d0c329-67b2-4229-b493-0b2415d41d29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:09.552432 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:09.600994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 735/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 03:42:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:42:09.894089 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2fa74e-b1bf-4068-ab5c-5cea1c89e3ba None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:42:09.895743 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-5e2fa74e-b1bf-4068-ab5c-5cea1c89e3ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:09.999351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2fa74e-b1bf-4068-ab5c-5cea1c89e3ba None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:42:10.047670 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:10.072178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f04d6be-a602-4d9d-8f64-29660d5d81f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:42:10.072779 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f04d6be-a602-4d9d-8f64-29660d5d81f0 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:10.073058 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f04d6be-a602-4d9d-8f64-29660d5d81f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:10.073329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f04d6be-a602-4d9d-8f64-29660d5d81f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:10.074969 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-9f04d6be-a602-4d9d-8f64-29660d5d81f0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:10.184146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f04d6be-a602-4d9d-8f64-29660d5d81f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:42:10.185507 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f04d6be-a602-4d9d-8f64-29660d5d81f0 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:42:10.204222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 735/1470] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:42:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:42:19.599452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=22 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:20.201329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=66 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:31.347193 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d720b628-b20f-4f07-a4f9-be2924518bf6 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:31.347620 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d720b628-b20f-4f07-a4f9-be2924518bf6 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:31.347767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d720b628-b20f-4f07-a4f9-be2924518bf6 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:31.349116 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 736/1471] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:42:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:42:31.354347 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144fa60b-b08d-46e2-b531-1997fb80c80f None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:31.354824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144fa60b-b08d-46e2-b531-1997fb80c80f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:31.354940 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144fa60b-b08d-46e2-b531-1997fb80c80f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:31.539017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-144fa60b-b08d-46e2-b531-1997fb80c80f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:42:31.540221 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-144fa60b-b08d-46e2-b531-1997fb80c80f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:31.633110 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 736/1472] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:42:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:42:31.943981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fd2ed8-9989-4b3c-9f14-b4e31d1aa7d9 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:31.944269 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fd2ed8-9989-4b3c-9f14-b4e31d1aa7d9 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:31.944545 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fd2ed8-9989-4b3c-9f14-b4e31d1aa7d9 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:31.945876 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 737/1473] 10.222.0.49 () {54 vars in 962 bytes} [Fri Sep 16 03:42:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:42:31.950807 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5441a09c-a979-4a0a-81ab-dfefcbfc39e0 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:31.951127 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5441a09c-a979-4a0a-81ab-dfefcbfc39e0 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:31.951408 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5441a09c-a979-4a0a-81ab-dfefcbfc39e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:32.026082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-5441a09c-a979-4a0a-81ab-dfefcbfc39e0 None None] MFA Rules not processed for user `05ceb9fc31c345ccabaee8f30a3982a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:42:32.027295 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-5441a09c-a979-4a0a-81ab-dfefcbfc39e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:32.048784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:32.102515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 737/1474] 10.222.0.49 () {58 vars in 1059 bytes} [Fri Sep 16 03:42:31 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:42:32.110763 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fd2ed8-9989-4b3c-9f14-b4e31d1aa7d9 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:42:32.112353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c4fd2ed8-9989-4b3c-9f14-b4e31d1aa7d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:32.219090 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fd2ed8-9989-4b3c-9f14-b4e31d1aa7d9 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:42:32.296101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f729ac52-247e-4a96-9ce3-9ad6bcd8d135 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ceb9fc31c345ccabaee8f30a3982a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:42:32.297476 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f729ac52-247e-4a96-9ce3-9ad6bcd8d135 service glance] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:32.297750 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f729ac52-247e-4a96-9ce3-9ad6bcd8d135 service glance] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:32.298015 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f729ac52-247e-4a96-9ce3-9ad6bcd8d135 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:32.299641 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f729ac52-247e-4a96-9ce3-9ad6bcd8d135 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:32.353764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:32.409311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f729ac52-247e-4a96-9ce3-9ad6bcd8d135 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:42:32.410756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f729ac52-247e-4a96-9ce3-9ad6bcd8d135 service glance] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:42:32.429524 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 738/1475] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:42:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:42:34.707963 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4b15ad-50e1-479d-8235-cf2f1caad0a1 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:34.708539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4b15ad-50e1-479d-8235-cf2f1caad0a1 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:34.708539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4b15ad-50e1-479d-8235-cf2f1caad0a1 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:34.709874 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 738/1476] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:42:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:42:34.715269 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b50ba49-68e5-418c-8edb-2e6034474a4d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:34.715693 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b50ba49-68e5-418c-8edb-2e6034474a4d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:34.715817 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b50ba49-68e5-418c-8edb-2e6034474a4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:34.781095 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7b50ba49-68e5-418c-8edb-2e6034474a4d None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:42:34.782307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7b50ba49-68e5-418c-8edb-2e6034474a4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:34.880068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 739/1477] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:42:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:42:35.164219 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4b15ad-50e1-479d-8235-cf2f1caad0a1 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:42:35.165665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ef4b15ad-50e1-479d-8235-cf2f1caad0a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:35.296358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4b15ad-50e1-479d-8235-cf2f1caad0a1 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:42:35.346772 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:35.390610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3ac650-891d-43c9-b282-141cd4621e1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:42:35.391240 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3ac650-891d-43c9-b282-141cd4621e1d admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:35.391519 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3ac650-891d-43c9-b282-141cd4621e1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:35.391789 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3ac650-891d-43c9-b282-141cd4621e1d admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:35.392426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3ac650-891d-43c9-b282-141cd4621e1d admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:42:35.394159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: WARNING py.warnings [None req-8d3ac650-891d-43c9-b282-141cd4621e1d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 03:42:35.394159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: warnings.warn(msg) Sep 16 03:42:35.394159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: Sep 16 03:42:35.395311 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3ac650-891d-43c9-b282-141cd4621e1d admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:42:35.413728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 739/1478] 10.222.0.49 () {56 vars in 1108 bytes} [Fri Sep 16 03:42:35 2022] GET /identity/v3/projects => generated 2553 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:42:37.638059 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d92b58-ab33-4871-a2d6-ac12bfff7bba None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:37.638606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d92b58-ab33-4871-a2d6-ac12bfff7bba None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:37.638606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d92b58-ab33-4871-a2d6-ac12bfff7bba None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:37.640480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 740/1479] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:42:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:42:37.646403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b010b0b5-9225-40f4-ba15-300eb95540da None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:37.646846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b010b0b5-9225-40f4-ba15-300eb95540da None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:37.646975 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b010b0b5-9225-40f4-ba15-300eb95540da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:37.713005 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-b010b0b5-9225-40f4-ba15-300eb95540da None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:42:37.714245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b010b0b5-9225-40f4-ba15-300eb95540da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:37.810280 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 740/1480] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:42:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:42:38.103178 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d92b58-ab33-4871-a2d6-ac12bfff7bba None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:42:38.104732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d4d92b58-ab33-4871-a2d6-ac12bfff7bba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:38.115076 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:38.250353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d92b58-ab33-4871-a2d6-ac12bfff7bba None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:42:38.330122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:38.363038 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51d17be-f76c-438b-b5e3-d3651547b17f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:42:38.363581 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51d17be-f76c-438b-b5e3-d3651547b17f service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:38.363862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51d17be-f76c-438b-b5e3-d3651547b17f service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:38.364137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51d17be-f76c-438b-b5e3-d3651547b17f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:38.365748 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f51d17be-f76c-438b-b5e3-d3651547b17f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:38.456787 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f51d17be-f76c-438b-b5e3-d3651547b17f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:42:38.458369 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f51d17be-f76c-438b-b5e3-d3651547b17f service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:42:38.476689 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 741/1481] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:42:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:42:40.811196 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c1808b-a3bc-47d2-a4e1-90aca1fb1411 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:40.811745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c1808b-a3bc-47d2-a4e1-90aca1fb1411 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:40.811865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c1808b-a3bc-47d2-a4e1-90aca1fb1411 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:40.813258 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 741/1482] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:42:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:42:40.818458 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e15007-9e91-4353-a0e1-21e1739c9adf None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:40.818928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e15007-9e91-4353-a0e1-21e1739c9adf None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:40.819052 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e15007-9e91-4353-a0e1-21e1739c9adf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:40.883998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-45e15007-9e91-4353-a0e1-21e1739c9adf None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:42:40.885185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-45e15007-9e91-4353-a0e1-21e1739c9adf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:40.930148 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:40.979066 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 742/1483] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:42:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:42:41.268612 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c1808b-a3bc-47d2-a4e1-90aca1fb1411 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:42:41.270124 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e6c1808b-a3bc-47d2-a4e1-90aca1fb1411 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:41.293330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:41.373626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c1808b-a3bc-47d2-a4e1-90aca1fb1411 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:42:41.447384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13f9e47-a8f6-4042-9515-d3cf622be693 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:42:41.447989 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13f9e47-a8f6-4042-9515-d3cf622be693 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:41.448285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13f9e47-a8f6-4042-9515-d3cf622be693 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:41.448544 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13f9e47-a8f6-4042-9515-d3cf622be693 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:41.450118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a13f9e47-a8f6-4042-9515-d3cf622be693 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:41.525034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:41.559843 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13f9e47-a8f6-4042-9515-d3cf622be693 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:42:41.561279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13f9e47-a8f6-4042-9515-d3cf622be693 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:42:41.579908 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 742/1484] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:42:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:42:43.770924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b7d4d0-3d89-47b4-904b-46b70945d356 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:43.771368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b7d4d0-3d89-47b4-904b-46b70945d356 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:43.771524 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b7d4d0-3d89-47b4-904b-46b70945d356 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:43.772819 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 743/1485] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:42:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:42:43.778093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b2d62b-8b21-4801-aaaa-e44c945f96cc None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:43.778496 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b2d62b-8b21-4801-aaaa-e44c945f96cc None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:43.778646 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b2d62b-8b21-4801-aaaa-e44c945f96cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:43.847916 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-d3b2d62b-8b21-4801-aaaa-e44c945f96cc None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:42:43.849120 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d3b2d62b-8b21-4801-aaaa-e44c945f96cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:43.943893 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 743/1486] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:42:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:42:44.229728 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b7d4d0-3d89-47b4-904b-46b70945d356 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:42:44.231286 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e4b7d4d0-3d89-47b4-904b-46b70945d356 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:44.335536 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b7d4d0-3d89-47b4-904b-46b70945d356 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:42:44.408367 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:44.409822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859b6e12-b6d1-4fe2-b7f2-572e896c33a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:42:44.410414 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859b6e12-b6d1-4fe2-b7f2-572e896c33a9 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:44.410690 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859b6e12-b6d1-4fe2-b7f2-572e896c33a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:44.410988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859b6e12-b6d1-4fe2-b7f2-572e896c33a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:44.412557 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-859b6e12-b6d1-4fe2-b7f2-572e896c33a9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:44.519863 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859b6e12-b6d1-4fe2-b7f2-572e896c33a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:42:44.521211 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859b6e12-b6d1-4fe2-b7f2-572e896c33a9 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:42:44.538859 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 744/1487] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:42:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:42:46.768831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d642fee2-7e2c-4ca4-8ecd-a7611ac0c0e7 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:46.769402 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d642fee2-7e2c-4ca4-8ecd-a7611ac0c0e7 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:46.769402 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d642fee2-7e2c-4ca4-8ecd-a7611ac0c0e7 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:46.770756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 744/1488] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:42:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:42:46.776205 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705b0e68-1925-4180-86e9-630ac81ff724 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:46.776639 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705b0e68-1925-4180-86e9-630ac81ff724 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:46.776737 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705b0e68-1925-4180-86e9-630ac81ff724 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:46.842002 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-705b0e68-1925-4180-86e9-630ac81ff724 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:42:46.843537 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-705b0e68-1925-4180-86e9-630ac81ff724 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:46.896516 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:46.943120 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 745/1489] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:42:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:42:47.239742 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d642fee2-7e2c-4ca4-8ecd-a7611ac0c0e7 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:42:47.241286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d642fee2-7e2c-4ca4-8ecd-a7611ac0c0e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:47.277768 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:47.344856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d642fee2-7e2c-4ca4-8ecd-a7611ac0c0e7 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:42:47.416566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8663642d-fed4-4315-8574-8175dcaa0bdc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:42:47.417162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8663642d-fed4-4315-8574-8175dcaa0bdc service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:47.417439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8663642d-fed4-4315-8574-8175dcaa0bdc service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:47.417710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8663642d-fed4-4315-8574-8175dcaa0bdc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:47.419307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-8663642d-fed4-4315-8574-8175dcaa0bdc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:47.499911 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:47.527751 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8663642d-fed4-4315-8574-8175dcaa0bdc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:42:47.529246 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8663642d-fed4-4315-8574-8175dcaa0bdc service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:42:47.549163 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 745/1490] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:42:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:42:50.152786 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4d312f-ae9b-4d7c-9b1a-7028fa31b45a None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:50.153362 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4d312f-ae9b-4d7c-9b1a-7028fa31b45a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:50.153362 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4d312f-ae9b-4d7c-9b1a-7028fa31b45a None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:50.154674 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 746/1491] 10.222.0.49 () {54 vars in 927 bytes} [Fri Sep 16 03:42:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:42:50.160828 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1925a1fd-bcfb-4f59-b231-fb9c1976de48 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:50.161300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1925a1fd-bcfb-4f59-b231-fb9c1976de48 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:50.161428 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1925a1fd-bcfb-4f59-b231-fb9c1976de48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:50.226008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-1925a1fd-bcfb-4f59-b231-fb9c1976de48 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:42:50.227245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1925a1fd-bcfb-4f59-b231-fb9c1976de48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:50.325691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 746/1492] 10.222.0.49 () {58 vars in 1024 bytes} [Fri Sep 16 03:42:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:42:50.621588 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4d312f-ae9b-4d7c-9b1a-7028fa31b45a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:42:50.627098 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3c4d312f-ae9b-4d7c-9b1a-7028fa31b45a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:50.726252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4d312f-ae9b-4d7c-9b1a-7028fa31b45a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:42:50.800244 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf36325-6fa9-4314-aa90-000d600ae7ef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:42:50.800841 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf36325-6fa9-4314-aa90-000d600ae7ef service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:42:50.801133 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf36325-6fa9-4314-aa90-000d600ae7ef service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:42:50.801402 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf36325-6fa9-4314-aa90-000d600ae7ef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:42:50.803029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3bf36325-6fa9-4314-aa90-000d600ae7ef service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:42:50.811326 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:42:50.913272 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf36325-6fa9-4314-aa90-000d600ae7ef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:42:50.914672 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf36325-6fa9-4314-aa90-000d600ae7ef service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:42:50.933626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 747/1493] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:42:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:43:00.324148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=82 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:43:00.931340 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=50 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:44:41.331799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a148e6e6-1a5d-443c-b7ce-9fbc98328e7d None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:44:41.332308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a148e6e6-1a5d-443c-b7ce-9fbc98328e7d None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:44:41.332416 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a148e6e6-1a5d-443c-b7ce-9fbc98328e7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:44:41.412746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-a148e6e6-1a5d-443c-b7ce-9fbc98328e7d None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:44:41.413930 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a148e6e6-1a5d-443c-b7ce-9fbc98328e7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:44:41.517580 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 747/1494] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:44:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:44:41.525579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435cf94f-2a85-42fe-ac2c-d37ba068841f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:44:41.526067 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435cf94f-2a85-42fe-ac2c-d37ba068841f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:44:41.526178 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435cf94f-2a85-42fe-ac2c-d37ba068841f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:44:41.546075 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-435cf94f-2a85-42fe-ac2c-d37ba068841f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:44:41.547542 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-435cf94f-2a85-42fe-ac2c-d37ba068841f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:44:41.555898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 748/1495] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:44:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:44:41.560984 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a148e6e6-1a5d-443c-b7ce-9fbc98328e7d None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:44:41.570521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a148e6e6-1a5d-443c-b7ce-9fbc98328e7d None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:44:41.575192 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c747d9d7-65ee-4da4-898b-02f04079745b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:44:41.575819 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c747d9d7-65ee-4da4-898b-02f04079745b admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:44:41.576172 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c747d9d7-65ee-4da4-898b-02f04079745b admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:44:41.576450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c747d9d7-65ee-4da4-898b-02f04079745b admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:44:41.577139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c747d9d7-65ee-4da4-898b-02f04079745b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:44:41.577399 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c747d9d7-65ee-4da4-898b-02f04079745b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:44:41.578445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: WARNING py.warnings [None req-c747d9d7-65ee-4da4-898b-02f04079745b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 16 03:44:41.578445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: warnings.warn(msg) Sep 16 03:44:41.578445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: Sep 16 03:44:41.579365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c747d9d7-65ee-4da4-898b-02f04079745b admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:44:41.590521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 748/1496] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:44:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:44:41.598823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435cf94f-2a85-42fe-ac2c-d37ba068841f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:44:41.608233 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435cf94f-2a85-42fe-ac2c-d37ba068841f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:44:41.612738 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a756c9-af40-4886-8b18-69fbb21b1ecd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:44:41.613327 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a756c9-af40-4886-8b18-69fbb21b1ecd admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:44:41.613603 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a756c9-af40-4886-8b18-69fbb21b1ecd admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:44:41.613868 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a756c9-af40-4886-8b18-69fbb21b1ecd admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:44:41.614497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a756c9-af40-4886-8b18-69fbb21b1ecd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:44:41.616253 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: WARNING py.warnings [None req-f3a756c9-af40-4886-8b18-69fbb21b1ecd admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 03:44:41.616253 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: warnings.warn(msg) Sep 16 03:44:41.616253 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: Sep 16 03:44:41.617436 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a756c9-af40-4886-8b18-69fbb21b1ecd admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:44:41.711804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 749/1497] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:44:41 2022] POST /identity/v3/projects => generated 377 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:44:41.716109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c747d9d7-65ee-4da4-898b-02f04079745b admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:44:41.726719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c747d9d7-65ee-4da4-898b-02f04079745b admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:44:41.775426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:44:41.782465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60111502-5995-4821-a3fa-e26eb5b0a780 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:44:41.785179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60111502-5995-4821-a3fa-e26eb5b0a780 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:44:41.785179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60111502-5995-4821-a3fa-e26eb5b0a780 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:44:41.785179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60111502-5995-4821-a3fa-e26eb5b0a780 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:44:41.785179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60111502-5995-4821-a3fa-e26eb5b0a780 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:44:41.785577 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: WARNING py.warnings [None req-60111502-5995-4821-a3fa-e26eb5b0a780 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 03:44:41.785577 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: warnings.warn(msg) Sep 16 03:44:41.785577 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: Sep 16 03:44:41.786407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60111502-5995-4821-a3fa-e26eb5b0a780 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:44:41.870450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 749/1498] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:44:41 2022] POST /identity/v3/users => generated 289 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:44:41.874608 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a756c9-af40-4886-8b18-69fbb21b1ecd admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:44:41.885183 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a756c9-af40-4886-8b18-69fbb21b1ecd admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:44:41.889828 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c1ae97-a62d-497e-b746-266416c9b4ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:44:41.890418 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c1ae97-a62d-497e-b746-266416c9b4ce admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:44:41.890698 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c1ae97-a62d-497e-b746-266416c9b4ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:44:41.891004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c1ae97-a62d-497e-b746-266416c9b4ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:44:41.891637 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c1ae97-a62d-497e-b746-266416c9b4ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:44:41.892673 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: WARNING py.warnings [None req-90c1ae97-a62d-497e-b746-266416c9b4ce admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 16 03:44:41.892673 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: warnings.warn(msg) Sep 16 03:44:41.892673 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: Sep 16 03:44:41.893539 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c1ae97-a62d-497e-b746-266416c9b4ce admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:44:41.906572 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 750/1499] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:44:41 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:44:41.910535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60111502-5995-4821-a3fa-e26eb5b0a780 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:44:41.921069 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60111502-5995-4821-a3fa-e26eb5b0a780 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:44:41.925893 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906f6993-f69f-4fa2-b646-c6f4d1a0f94d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:44:41.926480 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906f6993-f69f-4fa2-b646-c6f4d1a0f94d admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:44:41.926757 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906f6993-f69f-4fa2-b646-c6f4d1a0f94d admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:44:41.927062 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906f6993-f69f-4fa2-b646-c6f4d1a0f94d admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:44:41.927701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906f6993-f69f-4fa2-b646-c6f4d1a0f94d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:44:41.928737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: WARNING py.warnings [None req-906f6993-f69f-4fa2-b646-c6f4d1a0f94d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 16 03:44:41.928737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: warnings.warn(msg) Sep 16 03:44:41.928737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: Sep 16 03:44:41.929601 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906f6993-f69f-4fa2-b646-c6f4d1a0f94d admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:44:41.943649 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 750/1500] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:44:41 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:44:41.947635 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c1ae97-a62d-497e-b746-266416c9b4ce admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:44:41.957558 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c1ae97-a62d-497e-b746-266416c9b4ce admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:44:41.962118 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46d14b2-54f2-4ca2-bb39-ba385d6cf55f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:44:41.962760 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46d14b2-54f2-4ca2-bb39-ba385d6cf55f admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:44:41.963092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46d14b2-54f2-4ca2-bb39-ba385d6cf55f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:44:41.963375 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46d14b2-54f2-4ca2-bb39-ba385d6cf55f admin admin] PATH_INFO: `/v3/projects/a0d3a3ad8036497e9272f689823a015a/users/944b93f4dc4a40cbb52bc779f1ef6316/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:44:41.978392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46d14b2-54f2-4ca2-bb39-ba385d6cf55f admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0d3a3ad8036497e9272f689823a015a, user_id=944b93f4dc4a40cbb52bc779f1ef6316, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:44:41.979462 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: WARNING py.warnings [None req-e46d14b2-54f2-4ca2-bb39-ba385d6cf55f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 03:44:41.979462 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: warnings.warn(msg) Sep 16 03:44:41.979462 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: Sep 16 03:44:41.981424 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46d14b2-54f2-4ca2-bb39-ba385d6cf55f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:44:42.055075 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 751/1501] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:44:41 2022] PUT /identity/v3/projects/a0d3a3ad8036497e9272f689823a015a/users/944b93f4dc4a40cbb52bc779f1ef6316/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:44:42.074622 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a28e596-758c-403d-8914-4893f8743af9 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:44:42.074622 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a28e596-758c-403d-8914-4893f8743af9 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:44:42.074622 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a28e596-758c-403d-8914-4893f8743af9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:44:42.096606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-3a28e596-758c-403d-8914-4893f8743af9 None None] MFA Rules not processed for user `944b93f4dc4a40cbb52bc779f1ef6316`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:44:42.097801 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3a28e596-758c-403d-8914-4893f8743af9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:44:42.160598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 751/1502] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:44:42 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:44:42.174392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b4793c-14e1-4bac-9e3f-24eaefec992f None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:44:42.174678 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b4793c-14e1-4bac-9e3f-24eaefec992f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:44:42.174978 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b4793c-14e1-4bac-9e3f-24eaefec992f None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:44:42.176350 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 752/1503] 10.222.0.49 () {54 vars in 964 bytes} [Fri Sep 16 03:44:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:44:42.181257 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3552171a-8e43-4999-9626-1bc6deb7c8f9 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:44:42.181545 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3552171a-8e43-4999-9626-1bc6deb7c8f9 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:44:42.181825 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3552171a-8e43-4999-9626-1bc6deb7c8f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:44:42.241609 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-3552171a-8e43-4999-9626-1bc6deb7c8f9 None None] MFA Rules not processed for user `e6aeffd924734771bc5bf0d282c9052d`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:44:42.242811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3552171a-8e43-4999-9626-1bc6deb7c8f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:44:42.318239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 752/1504] 10.222.0.49 () {58 vars in 1061 bytes} [Fri Sep 16 03:44:42 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:44:42.328918 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b4793c-14e1-4bac-9e3f-24eaefec992f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:44:42.339175 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b4793c-14e1-4bac-9e3f-24eaefec992f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:44:42.343255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f97e8cf-067d-48e3-92da-612e508c8814 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:44:42.343842 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f97e8cf-067d-48e3-92da-612e508c8814 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:44:42.344119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f97e8cf-067d-48e3-92da-612e508c8814 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:44:42.344387 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f97e8cf-067d-48e3-92da-612e508c8814 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:44:42.352633 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f97e8cf-067d-48e3-92da-612e508c8814 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:44:42.354054 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f97e8cf-067d-48e3-92da-612e508c8814 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:44:42.362781 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 753/1505] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:44:42 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:44:42.407576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3c0ebe-5383-47a3-ac1e-19d12c32a6d1 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:44:42.407996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3c0ebe-5383-47a3-ac1e-19d12c32a6d1 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:44:42.408129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3c0ebe-5383-47a3-ac1e-19d12c32a6d1 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:44:42.409462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 753/1506] 10.222.0.49 () {52 vars in 830 bytes} [Fri Sep 16 03:44:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:44:42.413090 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f97e8cf-067d-48e3-92da-612e508c8814 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:44:42.477037 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f97e8cf-067d-48e3-92da-612e508c8814 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:44:42.482337 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f4963f-03fd-440e-9015-606fe0ab83c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:44:42.482947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f4963f-03fd-440e-9015-606fe0ab83c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:44:42.483228 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f4963f-03fd-440e-9015-606fe0ab83c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:44:42.483499 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f4963f-03fd-440e-9015-606fe0ab83c0 admin admin] PATH_INFO: `/v3/users/944b93f4dc4a40cbb52bc779f1ef6316` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:44:42.484625 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f4963f-03fd-440e-9015-606fe0ab83c0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=944b93f4dc4a40cbb52bc779f1ef6316)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:44:42.485626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: WARNING py.warnings [None req-f6f4963f-03fd-440e-9015-606fe0ab83c0 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 03:44:42.485626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: warnings.warn(msg) Sep 16 03:44:42.485626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: Sep 16 03:44:42.486812 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f4963f-03fd-440e-9015-606fe0ab83c0 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:44:42.765597 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f6f4963f-03fd-440e-9015-606fe0ab83c0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '944b93f4dc4a40cbb52bc779f1ef6316'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:44:42.790468 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f6f4963f-03fd-440e-9015-606fe0ab83c0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '944b93f4dc4a40cbb52bc779f1ef6316'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:44:42.864690 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f6f4963f-03fd-440e-9015-606fe0ab83c0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '944b93f4dc4a40cbb52bc779f1ef6316'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:44:42.865396 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f6f4963f-03fd-440e-9015-606fe0ab83c0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '944b93f4dc4a40cbb52bc779f1ef6316'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:44:42.875577 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:44:42.879982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f6f4963f-03fd-440e-9015-606fe0ab83c0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '944b93f4dc4a40cbb52bc779f1ef6316'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:44:42.888694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 754/1507] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:44:42 2022] DELETE /identity/v3/users/944b93f4dc4a40cbb52bc779f1ef6316 => generated 0 bytes in 476 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:44:42.893668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324542e2-a235-445c-9d58-562a894f6136 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:44:42.894006 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324542e2-a235-445c-9d58-562a894f6136 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:44:42.894420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324542e2-a235-445c-9d58-562a894f6136 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:44:42.914607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-324542e2-a235-445c-9d58-562a894f6136 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:44:42.915846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-324542e2-a235-445c-9d58-562a894f6136 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:44:42.925284 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:44:42.982788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 754/1508] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:44:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:44:42.993106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f4963f-03fd-440e-9015-606fe0ab83c0 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:44:42.994537 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f6f4963f-03fd-440e-9015-606fe0ab83c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:44:43.072790 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f4963f-03fd-440e-9015-606fe0ab83c0 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:44:43.077347 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774e2fd6-5810-4526-88d8-cf8c797d03cf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:44:43.077948 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774e2fd6-5810-4526-88d8-cf8c797d03cf service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:44:43.078231 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774e2fd6-5810-4526-88d8-cf8c797d03cf service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:44:43.078504 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774e2fd6-5810-4526-88d8-cf8c797d03cf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:44:43.087841 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774e2fd6-5810-4526-88d8-cf8c797d03cf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:44:43.089257 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774e2fd6-5810-4526-88d8-cf8c797d03cf service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:44:43.099735 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 755/1509] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:44:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:44:43.562966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324542e2-a235-445c-9d58-562a894f6136 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:44:43.564555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-324542e2-a235-445c-9d58-562a894f6136 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:44:43.582711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324542e2-a235-445c-9d58-562a894f6136 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:44:43.586190 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ecbfa2-2c8f-4259-832b-87cee0e4e0a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:44:43.586868 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ecbfa2-2c8f-4259-832b-87cee0e4e0a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:44:43.587185 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ecbfa2-2c8f-4259-832b-87cee0e4e0a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:44:43.587464 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ecbfa2-2c8f-4259-832b-87cee0e4e0a9 admin admin] PATH_INFO: `/v3/projects/a0d3a3ad8036497e9272f689823a015a` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:44:43.588948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ecbfa2-2c8f-4259-832b-87cee0e4e0a9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a0d3a3ad8036497e9272f689823a015a)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:44:43.590011 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: WARNING py.warnings [None req-89ecbfa2-2c8f-4259-832b-87cee0e4e0a9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 03:44:43.590011 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: warnings.warn(msg) Sep 16 03:44:43.590011 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: Sep 16 03:44:43.592180 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ecbfa2-2c8f-4259-832b-87cee0e4e0a9 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:44:43.720933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-89ecbfa2-2c8f-4259-832b-87cee0e4e0a9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a0d3a3ad8036497e9272f689823a015a', 'request_id': 'req-89ecbfa2-2c8f-4259-832b-87cee0e4e0a9'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:44:43.800116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-89ecbfa2-2c8f-4259-832b-87cee0e4e0a9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a0d3a3ad8036497e9272f689823a015a', 'request_id': 'req-89ecbfa2-2c8f-4259-832b-87cee0e4e0a9'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:44:43.807097 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-89ecbfa2-2c8f-4259-832b-87cee0e4e0a9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a0d3a3ad8036497e9272f689823a015a', 'request_id': 'req-89ecbfa2-2c8f-4259-832b-87cee0e4e0a9'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:44:43.825900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-89ecbfa2-2c8f-4259-832b-87cee0e4e0a9 admin admin] The token cache is being invalidate because project a0d3a3ad8036497e9272f689823a015a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:44:43.826310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-89ecbfa2-2c8f-4259-832b-87cee0e4e0a9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:44:43.827094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-89ecbfa2-2c8f-4259-832b-87cee0e4e0a9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:44:43.828780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 755/1510] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:44:43 2022] DELETE /identity/v3/projects/a0d3a3ad8036497e9272f689823a015a => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:44:53.095501 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=38 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:44:53.822294 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=47,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:46:12.347773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc8a1f9-e038-4549-897e-41a6fe929118 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:12.355519 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc8a1f9-e038-4549-897e-41a6fe929118 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:12.355519 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc8a1f9-e038-4549-897e-41a6fe929118 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:12.355519 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 756/1511] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 03:46:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:46:12.355819 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8763852-f8ae-432f-9888-c2bef5a6e7ae None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:12.355819 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8763852-f8ae-432f-9888-c2bef5a6e7ae None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:12.355819 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8763852-f8ae-432f-9888-c2bef5a6e7ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:12.376740 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-d8763852-f8ae-432f-9888-c2bef5a6e7ae None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:46:12.377969 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d8763852-f8ae-432f-9888-c2bef5a6e7ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:46:12.437350 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 756/1512] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 03:46:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:46:12.596827 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc8a1f9-e038-4549-897e-41a6fe929118 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:46:12.598675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1fc8a1f9-e038-4549-897e-41a6fe929118 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:46:12.708571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc8a1f9-e038-4549-897e-41a6fe929118 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:46:12.714038 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7400f4-a015-4041-af75-eac461f7d631 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:46:12.714638 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7400f4-a015-4041-af75-eac461f7d631 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:12.714939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7400f4-a015-4041-af75-eac461f7d631 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:12.715201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7400f4-a015-4041-af75-eac461f7d631 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:12.723164 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7400f4-a015-4041-af75-eac461f7d631 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:46:12.724726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7400f4-a015-4041-af75-eac461f7d631 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:46:12.734120 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 757/1513] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:46:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:46:13.812931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96328aaf-f804-431c-9efc-9b0ee0a040b4 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:13.813956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96328aaf-f804-431c-9efc-9b0ee0a040b4 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:13.814247 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96328aaf-f804-431c-9efc-9b0ee0a040b4 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:13.815686 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 757/1514] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 03:46:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:46:13.822374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0632626c-c8a1-40ed-9743-7d077ae0b408 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:13.823138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0632626c-c8a1-40ed-9743-7d077ae0b408 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:13.823138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0632626c-c8a1-40ed-9743-7d077ae0b408 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:13.843571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-0632626c-c8a1-40ed-9743-7d077ae0b408 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:46:13.844767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0632626c-c8a1-40ed-9743-7d077ae0b408 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:46:13.853083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 758/1515] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 03:46:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:46:14.007628 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96328aaf-f804-431c-9efc-9b0ee0a040b4 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:46:14.022391 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96328aaf-f804-431c-9efc-9b0ee0a040b4 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:46:14.027758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9439a793-4904-4faf-ac98-b6af5d1dc8a7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:46:14.028497 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9439a793-4904-4faf-ac98-b6af5d1dc8a7 service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:14.029184 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9439a793-4904-4faf-ac98-b6af5d1dc8a7 service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:14.029461 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9439a793-4904-4faf-ac98-b6af5d1dc8a7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:14.037558 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9439a793-4904-4faf-ac98-b6af5d1dc8a7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:46:14.039160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9439a793-4904-4faf-ac98-b6af5d1dc8a7 service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:46:14.048308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 758/1516] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:46:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:46:14.502407 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d5f82c-237a-4f36-a803-913f72e3223d None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:14.502718 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d5f82c-237a-4f36-a803-913f72e3223d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:14.503027 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d5f82c-237a-4f36-a803-913f72e3223d None None] PATH_INFO: `/v3` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:14.504292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 759/1517] 10.222.0.52 () {54 vars in 925 bytes} [Fri Sep 16 03:46:14 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:46:14.515650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0f4a0f-75de-41e0-ae4e-f06a3b03bed1 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:14.515940 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0f4a0f-75de-41e0-ae4e-f06a3b03bed1 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:14.516221 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0f4a0f-75de-41e0-ae4e-f06a3b03bed1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:14.565459 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-cd0f4a0f-75de-41e0-ae4e-f06a3b03bed1 None None] MFA Rules not processed for user `176da4679fcb46148da9289fccdad5c2`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:46:14.566643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-cd0f4a0f-75de-41e0-ae4e-f06a3b03bed1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:46:14.639849 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 759/1518] 10.222.0.52 () {58 vars in 1015 bytes} [Fri Sep 16 03:46:14 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:46:18.420526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d95205f-9f60-4491-9c9a-81834b585e93 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:18.420965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d95205f-9f60-4491-9c9a-81834b585e93 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:18.421113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d95205f-9f60-4491-9c9a-81834b585e93 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:18.422440 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 760/1519] 10.222.0.49 () {54 vars in 961 bytes} [Fri Sep 16 03:46:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:46:18.427616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1042e41c-87bb-4944-b0ad-df1054bb1e0b None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:18.428028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1042e41c-87bb-4944-b0ad-df1054bb1e0b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:18.428178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1042e41c-87bb-4944-b0ad-df1054bb1e0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:18.447917 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-1042e41c-87bb-4944-b0ad-df1054bb1e0b None None] MFA Rules not processed for user `176da4679fcb46148da9289fccdad5c2`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:46:18.452393 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1042e41c-87bb-4944-b0ad-df1054bb1e0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:46:18.458723 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 760/1520] 10.222.0.49 () {58 vars in 1058 bytes} [Fri Sep 16 03:46:18 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:46:18.467656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d95205f-9f60-4491-9c9a-81834b585e93 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:46:18.480939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d95205f-9f60-4491-9c9a-81834b585e93 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:46:18.485988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88457f8a-d7a6-48bf-a88b-67199244f5a0 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '176da4679fcb46148da9289fccdad5c2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:46:18.486576 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88457f8a-d7a6-48bf-a88b-67199244f5a0 service placement] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:18.486851 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88457f8a-d7a6-48bf-a88b-67199244f5a0 service placement] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:18.487150 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88457f8a-d7a6-48bf-a88b-67199244f5a0 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:18.495049 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88457f8a-d7a6-48bf-a88b-67199244f5a0 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:46:18.496622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88457f8a-d7a6-48bf-a88b-67199244f5a0 service placement] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:46:18.506738 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 761/1521] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:46:18 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:46:25.540251 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510891b9-954a-4bbc-af8c-d337ba76085c None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:25.540817 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510891b9-954a-4bbc-af8c-d337ba76085c None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:25.540817 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510891b9-954a-4bbc-af8c-d337ba76085c None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:25.542133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 761/1522] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 03:46:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:46:25.548230 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acaffe1f-fdff-494c-9b0f-a33a489d2bd6 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:25.548703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acaffe1f-fdff-494c-9b0f-a33a489d2bd6 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:25.549067 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acaffe1f-fdff-494c-9b0f-a33a489d2bd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:25.569083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-acaffe1f-fdff-494c-9b0f-a33a489d2bd6 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:46:25.570574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-acaffe1f-fdff-494c-9b0f-a33a489d2bd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:46:25.579423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 762/1523] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 03:46:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:46:25.734710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510891b9-954a-4bbc-af8c-d337ba76085c None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:46:25.749814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510891b9-954a-4bbc-af8c-d337ba76085c None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:46:25.755178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b704fd-d97c-44a6-9ddd-eb9b8acf6db7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:46:25.755769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b704fd-d97c-44a6-9ddd-eb9b8acf6db7 service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:25.756045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b704fd-d97c-44a6-9ddd-eb9b8acf6db7 service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:25.756314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b704fd-d97c-44a6-9ddd-eb9b8acf6db7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:25.765003 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b704fd-d97c-44a6-9ddd-eb9b8acf6db7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:46:25.766575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b704fd-d97c-44a6-9ddd-eb9b8acf6db7 service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:46:25.775697 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 762/1524] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:46:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:46:26.877176 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baee6979-c305-4995-b5ad-28ab3a612f6a None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:26.877755 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baee6979-c305-4995-b5ad-28ab3a612f6a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:26.877755 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baee6979-c305-4995-b5ad-28ab3a612f6a None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:26.879110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 763/1525] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 03:46:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:46:26.885171 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1de277-1fd3-4772-aff2-a728971e226f None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:26.885641 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1de277-1fd3-4772-aff2-a728971e226f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:26.885755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1de277-1fd3-4772-aff2-a728971e226f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:26.905410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-7c1de277-1fd3-4772-aff2-a728971e226f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:46:26.906599 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-7c1de277-1fd3-4772-aff2-a728971e226f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:46:26.914703 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 763/1526] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 03:46:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:46:27.097701 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baee6979-c305-4995-b5ad-28ab3a612f6a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:46:27.116308 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baee6979-c305-4995-b5ad-28ab3a612f6a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:46:27.124251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119eba85-e223-4bc1-a653-c47d663e3aa5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:46:27.124860 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119eba85-e223-4bc1-a653-c47d663e3aa5 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:27.125139 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119eba85-e223-4bc1-a653-c47d663e3aa5 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:27.125414 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119eba85-e223-4bc1-a653-c47d663e3aa5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:27.134190 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119eba85-e223-4bc1-a653-c47d663e3aa5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:46:27.135811 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119eba85-e223-4bc1-a653-c47d663e3aa5 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:46:27.148381 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 764/1527] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:46:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:46:31.055391 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7dffa1-75c7-43d6-97a0-e0ab65c00840 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:31.055997 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7dffa1-75c7-43d6-97a0-e0ab65c00840 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:31.055997 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7dffa1-75c7-43d6-97a0-e0ab65c00840 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:31.057308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 764/1528] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 03:46:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:46:31.064587 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca8fca0-13e8-4b3d-9178-d434b1011418 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:31.065015 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca8fca0-13e8-4b3d-9178-d434b1011418 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:31.065430 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca8fca0-13e8-4b3d-9178-d434b1011418 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:31.093498 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2ca8fca0-13e8-4b3d-9178-d434b1011418 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:46:31.095268 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2ca8fca0-13e8-4b3d-9178-d434b1011418 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:46:31.106314 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 765/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 03:46:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:46:31.405508 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7dffa1-75c7-43d6-97a0-e0ab65c00840 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:46:31.407716 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3f7dffa1-75c7-43d6-97a0-e0ab65c00840 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:46:31.438631 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:46:31.467815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7dffa1-75c7-43d6-97a0-e0ab65c00840 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:46:31.471324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbad5f5-83f2-4679-84b3-e9c55b9802d7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:46:31.471923 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbad5f5-83f2-4679-84b3-e9c55b9802d7 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:31.472202 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbad5f5-83f2-4679-84b3-e9c55b9802d7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:31.472472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbad5f5-83f2-4679-84b3-e9c55b9802d7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:31.481162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbad5f5-83f2-4679-84b3-e9c55b9802d7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:46:31.482614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbad5f5-83f2-4679-84b3-e9c55b9802d7 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:46:31.491683 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 765/1530] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:46:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:46:33.696806 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209d10ba-0dd2-40c6-9f7b-c4fbff8ed5f7 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:33.697356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209d10ba-0dd2-40c6-9f7b-c4fbff8ed5f7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:33.697356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209d10ba-0dd2-40c6-9f7b-c4fbff8ed5f7 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:33.698682 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 766/1531] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 03:46:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:46:33.704645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eec181b-8da2-4e1c-be4f-51532ad461dd None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:33.705044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eec181b-8da2-4e1c-be4f-51532ad461dd None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:33.705202 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eec181b-8da2-4e1c-be4f-51532ad461dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:33.724350 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-7eec181b-8da2-4e1c-be4f-51532ad461dd None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:46:33.725530 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-7eec181b-8da2-4e1c-be4f-51532ad461dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:46:33.733393 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 766/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 03:46:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:46:34.020726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209d10ba-0dd2-40c6-9f7b-c4fbff8ed5f7 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:46:34.026212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=68 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:46:34.032104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209d10ba-0dd2-40c6-9f7b-c4fbff8ed5f7 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:46:34.035598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cc52e2-4bd2-4e54-9efd-f82ac1ae77dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:46:34.036186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cc52e2-4bd2-4e54-9efd-f82ac1ae77dc service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:34.036463 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cc52e2-4bd2-4e54-9efd-f82ac1ae77dc service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:34.036735 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cc52e2-4bd2-4e54-9efd-f82ac1ae77dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:34.045842 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cc52e2-4bd2-4e54-9efd-f82ac1ae77dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:46:34.047291 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cc52e2-4bd2-4e54-9efd-f82ac1ae77dc service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:46:34.056819 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 767/1533] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:46:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:46:37.729058 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1916c316-7832-4330-9063-b44778dabe6a None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:37.729896 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1916c316-7832-4330-9063-b44778dabe6a None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:37.729896 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1916c316-7832-4330-9063-b44778dabe6a None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:37.731015 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 767/1534] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 03:46:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:46:37.736565 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a973bbc5-e045-46d4-b380-5beb8dd76ddb None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:37.737063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a973bbc5-e045-46d4-b380-5beb8dd76ddb None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:37.737166 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a973bbc5-e045-46d4-b380-5beb8dd76ddb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:37.762289 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a973bbc5-e045-46d4-b380-5beb8dd76ddb None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:46:37.764543 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a973bbc5-e045-46d4-b380-5beb8dd76ddb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:46:37.773797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 768/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 03:46:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:46:38.090034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1916c316-7832-4330-9063-b44778dabe6a None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:46:38.092012 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1916c316-7832-4330-9063-b44778dabe6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:46:38.148704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1916c316-7832-4330-9063-b44778dabe6a None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:46:38.152204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258b15d4-592e-490a-965a-36b305a02b21 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:46:38.152798 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258b15d4-592e-490a-965a-36b305a02b21 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:38.153077 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258b15d4-592e-490a-965a-36b305a02b21 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:38.153606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258b15d4-592e-490a-965a-36b305a02b21 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:38.162231 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258b15d4-592e-490a-965a-36b305a02b21 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:46:38.163677 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258b15d4-592e-490a-965a-36b305a02b21 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:46:38.172717 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 768/1536] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:46:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:46:40.336487 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48fc4eb-aa2b-4274-908a-4152f8e991d1 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:40.336941 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48fc4eb-aa2b-4274-908a-4152f8e991d1 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:40.336941 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48fc4eb-aa2b-4274-908a-4152f8e991d1 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:40.338282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 769/1537] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 03:46:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:46:40.343604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b32cc2-521c-4b4e-9251-5f56563a74d3 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:40.344040 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b32cc2-521c-4b4e-9251-5f56563a74d3 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:40.344148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b32cc2-521c-4b4e-9251-5f56563a74d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:40.363439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-88b32cc2-521c-4b4e-9251-5f56563a74d3 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:46:40.364608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-88b32cc2-521c-4b4e-9251-5f56563a74d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:46:40.373055 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 769/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 03:46:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:46:40.679496 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48fc4eb-aa2b-4274-908a-4152f8e991d1 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:46:40.681182 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c48fc4eb-aa2b-4274-908a-4152f8e991d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:46:40.694571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48fc4eb-aa2b-4274-908a-4152f8e991d1 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:46:40.698037 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8abd156-6289-494f-b185-11cf0fc58e6a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:46:40.698622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8abd156-6289-494f-b185-11cf0fc58e6a service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:46:40.698902 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8abd156-6289-494f-b185-11cf0fc58e6a service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:46:40.699198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8abd156-6289-494f-b185-11cf0fc58e6a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:46:40.708222 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8abd156-6289-494f-b185-11cf0fc58e6a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:46:40.709629 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8abd156-6289-494f-b185-11cf0fc58e6a service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:46:40.719442 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 770/1539] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:46:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:46:50.358035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=47 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:46:50.714681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=14 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:06.689458 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf09424-0eab-4865-a5f6-d0bac94f26a0 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:06.690145 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf09424-0eab-4865-a5f6-d0bac94f26a0 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:06.690145 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf09424-0eab-4865-a5f6-d0bac94f26a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:06.712399 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-bbf09424-0eab-4865-a5f6-d0bac94f26a0 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:06.714037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-bbf09424-0eab-4865-a5f6-d0bac94f26a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:06.727430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 770/1540] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:06.733729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2395d4ff-1f1a-4629-80e4-c68ee6a86788 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:06.734466 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2395d4ff-1f1a-4629-80e4-c68ee6a86788 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:06.734466 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2395d4ff-1f1a-4629-80e4-c68ee6a86788 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:06.766434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2395d4ff-1f1a-4629-80e4-c68ee6a86788 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:06.767904 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2395d4ff-1f1a-4629-80e4-c68ee6a86788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:06.778734 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 771/1541] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:06.784424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf09424-0eab-4865-a5f6-d0bac94f26a0 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:06.801735 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf09424-0eab-4865-a5f6-d0bac94f26a0 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:06.815810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc0d9b7-2ed3-4240-8f76-ddb428ebae9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:06.816455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc0d9b7-2ed3-4240-8f76-ddb428ebae9a admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:06.816734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc0d9b7-2ed3-4240-8f76-ddb428ebae9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:06.817007 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc0d9b7-2ed3-4240-8f76-ddb428ebae9a admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:06.817670 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc0d9b7-2ed3-4240-8f76-ddb428ebae9a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:48:06.817970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc0d9b7-2ed3-4240-8f76-ddb428ebae9a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:06.820104 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc0d9b7-2ed3-4240-8f76-ddb428ebae9a admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:06.837378 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 771/1542] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:48:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:48:06.846026 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2395d4ff-1f1a-4629-80e4-c68ee6a86788 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:06.859729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2395d4ff-1f1a-4629-80e4-c68ee6a86788 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:06.867696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6034ec46-6480-4879-bd80-aabc73a7bd9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:06.868328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6034ec46-6480-4879-bd80-aabc73a7bd9e admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:06.868608 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6034ec46-6480-4879-bd80-aabc73a7bd9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:06.868882 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6034ec46-6480-4879-bd80-aabc73a7bd9e admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:06.869601 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6034ec46-6480-4879-bd80-aabc73a7bd9e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:06.871614 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6034ec46-6480-4879-bd80-aabc73a7bd9e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:07.052946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 772/1543] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:06 2022] POST /identity/v3/projects => generated 385 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:07.057471 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc0d9b7-2ed3-4240-8f76-ddb428ebae9a admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:07.073038 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc0d9b7-2ed3-4240-8f76-ddb428ebae9a admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:07.118471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a78c01-2774-4aca-8a2a-4e8a575f67cb None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.118828 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a78c01-2774-4aca-8a2a-4e8a575f67cb None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.119152 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a78c01-2774-4aca-8a2a-4e8a575f67cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.151328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a8a78c01-2774-4aca-8a2a-4e8a575f67cb None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:07.152575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5260b6b0-8336-41c5-8e59-a91660e003ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:07.152843 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a8a78c01-2774-4aca-8a2a-4e8a575f67cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:07.153188 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5260b6b0-8336-41c5-8e59-a91660e003ad admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.153470 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5260b6b0-8336-41c5-8e59-a91660e003ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.153734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5260b6b0-8336-41c5-8e59-a91660e003ad admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.154384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5260b6b0-8336-41c5-8e59-a91660e003ad admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:07.157135 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5260b6b0-8336-41c5-8e59-a91660e003ad admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:07.170832 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 773/1544] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:48:07.172160 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d8b0c8-36df-48b8-9bd3-41514d6887b5 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.172445 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d8b0c8-36df-48b8-9bd3-41514d6887b5 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.172964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d8b0c8-36df-48b8-9bd3-41514d6887b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.201888 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-80d8b0c8-36df-48b8-9bd3-41514d6887b5 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:07.203407 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-80d8b0c8-36df-48b8-9bd3-41514d6887b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:07.217816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 774/1545] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:07.219174 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4858eb4-8048-4378-8ef5-a6f3a40a332f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.219461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4858eb4-8048-4378-8ef5-a6f3a40a332f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.219921 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4858eb4-8048-4378-8ef5-a6f3a40a332f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.249690 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-c4858eb4-8048-4378-8ef5-a6f3a40a332f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:07.250997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c4858eb4-8048-4378-8ef5-a6f3a40a332f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:07.258981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 775/1546] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:07.260168 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab08399-dc1a-49f3-857f-1c58b1e5ac42 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.260452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab08399-dc1a-49f3-857f-1c58b1e5ac42 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.260729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab08399-dc1a-49f3-857f-1c58b1e5ac42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.280938 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-cab08399-dc1a-49f3-857f-1c58b1e5ac42 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:07.282121 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cab08399-dc1a-49f3-857f-1c58b1e5ac42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:07.289813 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 776/1547] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:07.290966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eefccdb-6771-4e36-80ab-c328e2fdea44 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.291251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eefccdb-6771-4e36-80ab-c328e2fdea44 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.291526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eefccdb-6771-4e36-80ab-c328e2fdea44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.310999 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 772/1548] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:07 2022] POST /identity/v3/users => generated 293 bytes in 251 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:07.310999 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1185c9-ebee-4e59-a9ac-d43b7688fca1 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.310999 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1185c9-ebee-4e59-a9ac-d43b7688fca1 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.310999 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1185c9-ebee-4e59-a9ac-d43b7688fca1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.315882 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-6eefccdb-6771-4e36-80ab-c328e2fdea44 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:07.317137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6eefccdb-6771-4e36-80ab-c328e2fdea44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:07.338593 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-ad1185c9-ebee-4e59-a9ac-d43b7688fca1 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:07.338593 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ad1185c9-ebee-4e59-a9ac-d43b7688fca1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:07.338835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 777/1549] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:07.338835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebed997d-06b9-4ddc-9c20-4a414bb6d538 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.338835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebed997d-06b9-4ddc-9c20-4a414bb6d538 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.338835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebed997d-06b9-4ddc-9c20-4a414bb6d538 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.346430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 773/1550] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:07.347683 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f827c9-1059-4804-a84e-072eddcae8d2 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.348097 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f827c9-1059-4804-a84e-072eddcae8d2 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.348380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f827c9-1059-4804-a84e-072eddcae8d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.354213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ebed997d-06b9-4ddc-9c20-4a414bb6d538 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:07.355567 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ebed997d-06b9-4ddc-9c20-4a414bb6d538 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:07.364134 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 778/1551] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:07.365364 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd0d154-f474-4122-abe0-a1aa75ddc80d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.365650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd0d154-f474-4122-abe0-a1aa75ddc80d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.365925 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd0d154-f474-4122-abe0-a1aa75ddc80d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.368934 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-56f827c9-1059-4804-a84e-072eddcae8d2 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:07.370139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-56f827c9-1059-4804-a84e-072eddcae8d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:07.378326 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 774/1552] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:07.379159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f827c9-1059-4804-a84e-072eddcae8d2 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:07.386535 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ecd0d154-f474-4122-abe0-a1aa75ddc80d None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:07.390019 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ecd0d154-f474-4122-abe0-a1aa75ddc80d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:07.392743 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f827c9-1059-4804-a84e-072eddcae8d2 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:07.396496 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 779/1553] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:07.397324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e56b0b-f58b-4d2c-99ea-66854944d82a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:07.398000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e56b0b-f58b-4d2c-99ea-66854944d82a admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.398284 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e56b0b-f58b-4d2c-99ea-66854944d82a admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.398558 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e56b0b-f58b-4d2c-99ea-66854944d82a admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.399239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e56b0b-f58b-4d2c-99ea-66854944d82a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:48:07.399545 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e56b0b-f58b-4d2c-99ea-66854944d82a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:07.399744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd0d154-f474-4122-abe0-a1aa75ddc80d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:07.401138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e56b0b-f58b-4d2c-99ea-66854944d82a admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:07.410358 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd0d154-f474-4122-abe0-a1aa75ddc80d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:07.415007 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 775/1554] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:48:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:48:07.415935 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e56b0b-f58b-4d2c-99ea-66854944d82a admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:07.419130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e07e06c-54d7-4581-b213-f7cf5b6fd923 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:07.419821 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e07e06c-54d7-4581-b213-f7cf5b6fd923 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.420115 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e07e06c-54d7-4581-b213-f7cf5b6fd923 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.420389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e07e06c-54d7-4581-b213-f7cf5b6fd923 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.421136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e07e06c-54d7-4581-b213-f7cf5b6fd923 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:48:07.421441 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e07e06c-54d7-4581-b213-f7cf5b6fd923 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:07.422590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: WARNING py.warnings [None req-2e07e06c-54d7-4581-b213-f7cf5b6fd923 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 16 03:48:07.422590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: warnings.warn(msg) Sep 16 03:48:07.422590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: Sep 16 03:48:07.429941 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e07e06c-54d7-4581-b213-f7cf5b6fd923 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:07.430066 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e56b0b-f58b-4d2c-99ea-66854944d82a admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:07.431341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f6b07b-30fe-43b4-8e32-d991a365bc04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:07.431936 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f6b07b-30fe-43b4-8e32-d991a365bc04 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.432227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f6b07b-30fe-43b4-8e32-d991a365bc04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.432640 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f6b07b-30fe-43b4-8e32-d991a365bc04 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.433291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f6b07b-30fe-43b4-8e32-d991a365bc04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:07.434869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f6b07b-30fe-43b4-8e32-d991a365bc04 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:07.442748 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 780/1555] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:48:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:48:07.442748 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e545e40-964b-4c37-982b-0e47d1e6c8cf None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.442748 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e545e40-964b-4c37-982b-0e47d1e6c8cf None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.442748 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e545e40-964b-4c37-982b-0e47d1e6c8cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.446279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 776/1556] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:07.447447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b066bead-bde7-48ab-ac9d-bb5e6364f5ab None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.447731 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b066bead-bde7-48ab-ac9d-bb5e6364f5ab None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.448005 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b066bead-bde7-48ab-ac9d-bb5e6364f5ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.462481 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2e545e40-964b-4c37-982b-0e47d1e6c8cf None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:07.463725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2e545e40-964b-4c37-982b-0e47d1e6c8cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:07.467422 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-b066bead-bde7-48ab-ac9d-bb5e6364f5ab None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:07.468605 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b066bead-bde7-48ab-ac9d-bb5e6364f5ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:07.471909 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 781/1557] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:07.473160 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52df67d7-b951-4fa6-8228-bcd3616ce038 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.473531 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52df67d7-b951-4fa6-8228-bcd3616ce038 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.473812 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52df67d7-b951-4fa6-8228-bcd3616ce038 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.480105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 777/1558] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:07.481308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f870fe88-400c-4880-b868-8995505d4281 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.481597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f870fe88-400c-4880-b868-8995505d4281 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.481874 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f870fe88-400c-4880-b868-8995505d4281 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.494685 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-52df67d7-b951-4fa6-8228-bcd3616ce038 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:07.496127 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-52df67d7-b951-4fa6-8228-bcd3616ce038 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:07.501754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-f870fe88-400c-4880-b868-8995505d4281 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:07.502927 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-f870fe88-400c-4880-b868-8995505d4281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:07.505060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 782/1559] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:07.506277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afcb65c-02f4-4fe5-9f2b-31b5f88256a8 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.506615 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afcb65c-02f4-4fe5-9f2b-31b5f88256a8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.506891 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afcb65c-02f4-4fe5-9f2b-31b5f88256a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.514825 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 778/1560] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:07.515844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f870fe88-400c-4880-b868-8995505d4281 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:07.525148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f870fe88-400c-4880-b868-8995505d4281 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:07.527931 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-4afcb65c-02f4-4fe5-9f2b-31b5f88256a8 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:07.529128 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4afcb65c-02f4-4fe5-9f2b-31b5f88256a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:07.529952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cbc335-7d0a-4e6f-8bc5-3a8487f902c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:07.531101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cbc335-7d0a-4e6f-8bc5-3a8487f902c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.531386 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cbc335-7d0a-4e6f-8bc5-3a8487f902c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.531658 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cbc335-7d0a-4e6f-8bc5-3a8487f902c3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.532289 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7cbc335-7d0a-4e6f-8bc5-3a8487f902c3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:07.533317 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: WARNING py.warnings [None req-c7cbc335-7d0a-4e6f-8bc5-3a8487f902c3 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 03:48:07.533317 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: warnings.warn(msg) Sep 16 03:48:07.533317 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: Sep 16 03:48:07.534489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7cbc335-7d0a-4e6f-8bc5-3a8487f902c3 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:07.537731 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 783/1561] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:07.538477 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afcb65c-02f4-4fe5-9f2b-31b5f88256a8 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:07.548797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afcb65c-02f4-4fe5-9f2b-31b5f88256a8 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:07.554201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78616ad3-9d8b-43e9-8e4d-a0e05d132ca1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:07.554799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78616ad3-9d8b-43e9-8e4d-a0e05d132ca1 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.555111 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78616ad3-9d8b-43e9-8e4d-a0e05d132ca1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.555410 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78616ad3-9d8b-43e9-8e4d-a0e05d132ca1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.556094 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78616ad3-9d8b-43e9-8e4d-a0e05d132ca1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:07.558011 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78616ad3-9d8b-43e9-8e4d-a0e05d132ca1 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:07.612824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 779/1562] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:07 2022] POST /identity/v3/projects => generated 393 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:07.613725 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cbc335-7d0a-4e6f-8bc5-3a8487f902c3 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:07.636095 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 784/1563] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:07 2022] POST /identity/v3/projects => generated 371 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:07.636924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78616ad3-9d8b-43e9-8e4d-a0e05d132ca1 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:07.673993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:07.678016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cbc335-7d0a-4e6f-8bc5-3a8487f902c3 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:07.702783 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78616ad3-9d8b-43e9-8e4d-a0e05d132ca1 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:07.707688 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fde6735-5ad4-481c-8ef2-9bb5cbc59c4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:07.708865 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fde6735-5ad4-481c-8ef2-9bb5cbc59c4d admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.708865 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fde6735-5ad4-481c-8ef2-9bb5cbc59c4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.708865 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fde6735-5ad4-481c-8ef2-9bb5cbc59c4d admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.709495 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fde6735-5ad4-481c-8ef2-9bb5cbc59c4d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:48:07.709798 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fde6735-5ad4-481c-8ef2-9bb5cbc59c4d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:07.711525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fde6735-5ad4-481c-8ef2-9bb5cbc59c4d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:07.724305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 785/1564] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:48:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:48:07.725107 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fde6735-5ad4-481c-8ef2-9bb5cbc59c4d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:07.736640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fde6735-5ad4-481c-8ef2-9bb5cbc59c4d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:07.741245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8142dda3-d0e3-4aa6-9377-dbfc6ad3ce82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:07.741826 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8142dda3-d0e3-4aa6-9377-dbfc6ad3ce82 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.742103 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8142dda3-d0e3-4aa6-9377-dbfc6ad3ce82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.742320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de148a96-ddf2-4ebb-993b-35171fc85070 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:07.742755 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8142dda3-d0e3-4aa6-9377-dbfc6ad3ce82 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.742966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de148a96-ddf2-4ebb-993b-35171fc85070 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.743085 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8142dda3-d0e3-4aa6-9377-dbfc6ad3ce82 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:48:07.743269 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de148a96-ddf2-4ebb-993b-35171fc85070 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.743376 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8142dda3-d0e3-4aa6-9377-dbfc6ad3ce82 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:07.743535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de148a96-ddf2-4ebb-993b-35171fc85070 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.744189 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de148a96-ddf2-4ebb-993b-35171fc85070 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:07.744859 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8142dda3-d0e3-4aa6-9377-dbfc6ad3ce82 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:07.745772 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de148a96-ddf2-4ebb-993b-35171fc85070 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:07.755628 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 780/1565] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:07 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:07.756432 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de148a96-ddf2-4ebb-993b-35171fc85070 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:07.758963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 786/1566] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:48:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:48:07.758963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8142dda3-d0e3-4aa6-9377-dbfc6ad3ce82 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:07.769351 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8142dda3-d0e3-4aa6-9377-dbfc6ad3ce82 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:07.770307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de148a96-ddf2-4ebb-993b-35171fc85070 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:07.773941 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f81db1d-9463-4c0f-aa91-6f14e4dfa3ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:07.774537 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f81db1d-9463-4c0f-aa91-6f14e4dfa3ba admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.774819 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f81db1d-9463-4c0f-aa91-6f14e4dfa3ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.775019 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6580a5a8-d664-4223-9d46-596013d4ecaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:07.775146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f81db1d-9463-4c0f-aa91-6f14e4dfa3ba admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.775622 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6580a5a8-d664-4223-9d46-596013d4ecaf admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.775752 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f81db1d-9463-4c0f-aa91-6f14e4dfa3ba admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:48:07.775922 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6580a5a8-d664-4223-9d46-596013d4ecaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.776055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f81db1d-9463-4c0f-aa91-6f14e4dfa3ba admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:07.776220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6580a5a8-d664-4223-9d46-596013d4ecaf admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.776852 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6580a5a8-d664-4223-9d46-596013d4ecaf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:48:07.777154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6580a5a8-d664-4223-9d46-596013d4ecaf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:07.777632 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f81db1d-9463-4c0f-aa91-6f14e4dfa3ba admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:07.778756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6580a5a8-d664-4223-9d46-596013d4ecaf admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:07.784852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:07.789876 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 787/1567] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:48:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:48:07.790713 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f81db1d-9463-4c0f-aa91-6f14e4dfa3ba admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:07.792777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 781/1568] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:48:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:48:07.796370 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6580a5a8-d664-4223-9d46-596013d4ecaf admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:07.805259 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f81db1d-9463-4c0f-aa91-6f14e4dfa3ba admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:07.806745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6580a5a8-d664-4223-9d46-596013d4ecaf admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:07.809972 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa37fcb8-c955-4213-8877-0f12e3e070aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:07.810567 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa37fcb8-c955-4213-8877-0f12e3e070aa admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.810848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa37fcb8-c955-4213-8877-0f12e3e070aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.811184 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa37fcb8-c955-4213-8877-0f12e3e070aa admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.811467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdcae7a8-312f-4043-b0c0-a5f94212b9e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:07.811816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa37fcb8-c955-4213-8877-0f12e3e070aa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:48:07.812147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa37fcb8-c955-4213-8877-0f12e3e070aa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:07.812421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcae7a8-312f-4043-b0c0-a5f94212b9e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.812717 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcae7a8-312f-4043-b0c0-a5f94212b9e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.812995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcae7a8-312f-4043-b0c0-a5f94212b9e7 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.813636 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdcae7a8-312f-4043-b0c0-a5f94212b9e7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:07.813741 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa37fcb8-c955-4213-8877-0f12e3e070aa admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:07.815710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdcae7a8-312f-4043-b0c0-a5f94212b9e7 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:07.827257 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 788/1569] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:48:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:48:07.828281 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa37fcb8-c955-4213-8877-0f12e3e070aa admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:07.838015 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa37fcb8-c955-4213-8877-0f12e3e070aa admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:07.843539 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ce278f-9a7f-469e-8f0b-2af58920a698 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:07.844185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ce278f-9a7f-469e-8f0b-2af58920a698 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.844466 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ce278f-9a7f-469e-8f0b-2af58920a698 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.844743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ce278f-9a7f-469e-8f0b-2af58920a698 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.845354 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ce278f-9a7f-469e-8f0b-2af58920a698 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:07.846362 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: WARNING py.warnings [None req-a3ce278f-9a7f-469e-8f0b-2af58920a698 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 03:48:07.846362 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: warnings.warn(msg) Sep 16 03:48:07.846362 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: Sep 16 03:48:07.847589 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ce278f-9a7f-469e-8f0b-2af58920a698 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:07.961136 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 782/1570] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:07 2022] POST /identity/v3/users => generated 297 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:07.963678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdcae7a8-312f-4043-b0c0-a5f94212b9e7 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:07.972195 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdcae7a8-312f-4043-b0c0-a5f94212b9e7 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:07.976949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1fff0d-745a-4475-8811-092db8898340 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:07.977544 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1fff0d-745a-4475-8811-092db8898340 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:07.977824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1fff0d-745a-4475-8811-092db8898340 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:07.978098 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1fff0d-745a-4475-8811-092db8898340 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:07.978720 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1fff0d-745a-4475-8811-092db8898340 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:07.980670 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1fff0d-745a-4475-8811-092db8898340 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:07.989927 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 789/1571] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:07 2022] POST /identity/v3/users => generated 286 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:07.999613 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ce278f-9a7f-469e-8f0b-2af58920a698 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:08.010697 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ce278f-9a7f-469e-8f0b-2af58920a698 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:08.016056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c231dd4-5684-4006-8855-a0c66fb635d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:08.016704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c231dd4-5684-4006-8855-a0c66fb635d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:08.016983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c231dd4-5684-4006-8855-a0c66fb635d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:08.017255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c231dd4-5684-4006-8855-a0c66fb635d5 admin admin] PATH_INFO: `/v3/projects/8ee4a40c6c264553a14e0eff367da4a0/users/ff7df9be8475405eba924b59a7bd6bb0/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:08.033349 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c231dd4-5684-4006-8855-a0c66fb635d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ee4a40c6c264553a14e0eff367da4a0, user_id=ff7df9be8475405eba924b59a7bd6bb0, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:08.036004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c231dd4-5684-4006-8855-a0c66fb635d5 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:08.075068 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 783/1572] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:07 2022] POST /identity/v3/projects => generated 377 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:08.076159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1fff0d-745a-4475-8811-092db8898340 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:08.125646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 790/1573] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:07 2022] PUT /identity/v3/projects/8ee4a40c6c264553a14e0eff367da4a0/users/ff7df9be8475405eba924b59a7bd6bb0/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:08.127290 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c231dd4-5684-4006-8855-a0c66fb635d5 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:08.136597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1fff0d-745a-4475-8811-092db8898340 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:08.192033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:08.192965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c231dd4-5684-4006-8855-a0c66fb635d5 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:08.197644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0900dae-81d5-45c8-9bc9-e7bb274d898a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:08.198256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0900dae-81d5-45c8-9bc9-e7bb274d898a admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:08.198539 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0900dae-81d5-45c8-9bc9-e7bb274d898a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:08.198815 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0900dae-81d5-45c8-9bc9-e7bb274d898a admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:08.199089 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b44107-60f8-433d-961a-01fa05098f51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:08.199483 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0900dae-81d5-45c8-9bc9-e7bb274d898a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:08.199700 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b44107-60f8-433d-961a-01fa05098f51 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:08.200007 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b44107-60f8-433d-961a-01fa05098f51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:08.200251 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b44107-60f8-433d-961a-01fa05098f51 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:08.200883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b44107-60f8-433d-961a-01fa05098f51 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:08.201400 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0900dae-81d5-45c8-9bc9-e7bb274d898a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:08.203364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b44107-60f8-433d-961a-01fa05098f51 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:08.279345 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 791/1574] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:08 2022] POST /identity/v3/projects => generated 377 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:08.279345 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0900dae-81d5-45c8-9bc9-e7bb274d898a admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:08.298612 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 784/1575] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:08 2022] POST /identity/v3/projects => generated 389 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:08.303409 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b44107-60f8-433d-961a-01fa05098f51 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:08.344956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0900dae-81d5-45c8-9bc9-e7bb274d898a admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:08.367307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b44107-60f8-433d-961a-01fa05098f51 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:08.371925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49016be-ca3a-46b2-81e9-3474645578c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:08.372526 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49016be-ca3a-46b2-81e9-3474645578c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:08.372803 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49016be-ca3a-46b2-81e9-3474645578c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:08.373072 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49016be-ca3a-46b2-81e9-3474645578c3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:08.374263 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49016be-ca3a-46b2-81e9-3474645578c3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:08.376869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49016be-ca3a-46b2-81e9-3474645578c3 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:08.377953 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:08.401323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48ac2f0-c877-418d-8ed8-e8661140e572 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:08.401937 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48ac2f0-c877-418d-8ed8-e8661140e572 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:08.402215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48ac2f0-c877-418d-8ed8-e8661140e572 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:08.402485 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48ac2f0-c877-418d-8ed8-e8661140e572 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:08.403134 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48ac2f0-c877-418d-8ed8-e8661140e572 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:08.405597 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48ac2f0-c877-418d-8ed8-e8661140e572 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:08.518459 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 785/1576] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:08 2022] POST /identity/v3/projects => generated 405 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:08.518459 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49016be-ca3a-46b2-81e9-3474645578c3 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:08.527967 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 792/1577] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:08 2022] POST /identity/v3/projects => generated 377 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:08.532878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48ac2f0-c877-418d-8ed8-e8661140e572 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:08.590351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49016be-ca3a-46b2-81e9-3474645578c3 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:08.603823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48ac2f0-c877-418d-8ed8-e8661140e572 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:08.609047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4968f816-dc5e-48e8-a5ba-dffdf29da32f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:08.609658 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4968f816-dc5e-48e8-a5ba-dffdf29da32f admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:08.609934 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4968f816-dc5e-48e8-a5ba-dffdf29da32f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:08.610214 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4968f816-dc5e-48e8-a5ba-dffdf29da32f admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:08.611152 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4968f816-dc5e-48e8-a5ba-dffdf29da32f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:08.612737 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4968f816-dc5e-48e8-a5ba-dffdf29da32f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:08.623069 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 793/1578] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:08 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:08.624029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4968f816-dc5e-48e8-a5ba-dffdf29da32f admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:08.633374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4968f816-dc5e-48e8-a5ba-dffdf29da32f admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:08.638061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c229ce24-a0c0-438c-b861-3cccb90d5d1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:08.639305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c229ce24-a0c0-438c-b861-3cccb90d5d1c admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:08.639305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c229ce24-a0c0-438c-b861-3cccb90d5d1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:08.639305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c229ce24-a0c0-438c-b861-3cccb90d5d1c admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:08.639968 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c229ce24-a0c0-438c-b861-3cccb90d5d1c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:08.641991 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c229ce24-a0c0-438c-b861-3cccb90d5d1c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:08.650180 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4159715-cb49-4638-9592-ce3aaa5f5c67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:08.650814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4159715-cb49-4638-9592-ce3aaa5f5c67 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:08.651118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4159715-cb49-4638-9592-ce3aaa5f5c67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:08.651397 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4159715-cb49-4638-9592-ce3aaa5f5c67 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:08.652043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4159715-cb49-4638-9592-ce3aaa5f5c67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:08.653637 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4159715-cb49-4638-9592-ce3aaa5f5c67 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:08.664497 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 786/1579] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:08 2022] GET /identity/v3/roles => generated 1420 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:08.665687 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccba257c-3696-446b-bbd5-f8689640c47f None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:08.666055 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccba257c-3696-446b-bbd5-f8689640c47f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:08.666275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccba257c-3696-446b-bbd5-f8689640c47f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:08.685291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-ccba257c-3696-446b-bbd5-f8689640c47f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:08.686735 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ccba257c-3696-446b-bbd5-f8689640c47f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:08.694378 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 787/1580] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:08.698436 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccba257c-3696-446b-bbd5-f8689640c47f None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:08.699531 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:08.705528 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccba257c-3696-446b-bbd5-f8689640c47f None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:08.710092 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f54b517-e87e-47c6-9d8b-932a2d9e08fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:08.710689 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f54b517-e87e-47c6-9d8b-932a2d9e08fa admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:08.711000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f54b517-e87e-47c6-9d8b-932a2d9e08fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:08.711280 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f54b517-e87e-47c6-9d8b-932a2d9e08fa admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:08.711913 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f54b517-e87e-47c6-9d8b-932a2d9e08fa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:08.713820 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f54b517-e87e-47c6-9d8b-932a2d9e08fa admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:08.721604 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 794/1581] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:08 2022] POST /identity/v3/users => generated 289 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:08.721604 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c229ce24-a0c0-438c-b861-3cccb90d5d1c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:08.731706 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c229ce24-a0c0-438c-b861-3cccb90d5d1c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:08.736668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67651d32-f5a7-4dd5-8100-99c83e934dc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:08.737290 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67651d32-f5a7-4dd5-8100-99c83e934dc9 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:08.737566 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67651d32-f5a7-4dd5-8100-99c83e934dc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:08.737836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67651d32-f5a7-4dd5-8100-99c83e934dc9 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:08.738448 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67651d32-f5a7-4dd5-8100-99c83e934dc9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:08.740725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67651d32-f5a7-4dd5-8100-99c83e934dc9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:08.793640 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 788/1582] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:08 2022] POST /identity/v3/users => generated 289 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:08.794533 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f54b517-e87e-47c6-9d8b-932a2d9e08fa admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:08.804548 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f54b517-e87e-47c6-9d8b-932a2d9e08fa admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:08.809021 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cee8ea-bff1-4bb8-80f4-f45b33e10423 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:08.809593 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cee8ea-bff1-4bb8-80f4-f45b33e10423 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:08.809866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cee8ea-bff1-4bb8-80f4-f45b33e10423 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:08.810137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cee8ea-bff1-4bb8-80f4-f45b33e10423 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:08.810741 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cee8ea-bff1-4bb8-80f4-f45b33e10423 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:08.813342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cee8ea-bff1-4bb8-80f4-f45b33e10423 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:08.846336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 795/1583] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:08 2022] POST /identity/v3/users => generated 295 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:08.851448 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67651d32-f5a7-4dd5-8100-99c83e934dc9 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:08.856564 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67651d32-f5a7-4dd5-8100-99c83e934dc9 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:08.861884 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff08488-cdef-4583-8ff2-a6c593163436 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:08.862452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff08488-cdef-4583-8ff2-a6c593163436 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:08.862728 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff08488-cdef-4583-8ff2-a6c593163436 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:08.863031 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff08488-cdef-4583-8ff2-a6c593163436 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:08.863676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff08488-cdef-4583-8ff2-a6c593163436 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:08.866392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff08488-cdef-4583-8ff2-a6c593163436 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:08.914148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 789/1584] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:08 2022] POST /identity/v3/users => generated 303 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:08.915091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cee8ea-bff1-4bb8-80f4-f45b33e10423 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:08.924864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cee8ea-bff1-4bb8-80f4-f45b33e10423 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:08.929794 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3725f666-0ceb-4c11-8b26-218ca1ef2125 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:08.930385 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3725f666-0ceb-4c11-8b26-218ca1ef2125 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:08.930668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3725f666-0ceb-4c11-8b26-218ca1ef2125 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:08.930969 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3725f666-0ceb-4c11-8b26-218ca1ef2125 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:08.931605 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3725f666-0ceb-4c11-8b26-218ca1ef2125 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:08.933213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3725f666-0ceb-4c11-8b26-218ca1ef2125 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:08.943651 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 790/1585] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:08 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:08.944441 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3725f666-0ceb-4c11-8b26-218ca1ef2125 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:08.954591 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3725f666-0ceb-4c11-8b26-218ca1ef2125 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:08.959292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e227aae-5428-4eb8-803e-2f6f9b1b94ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:08.959869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e227aae-5428-4eb8-803e-2f6f9b1b94ef admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:08.960150 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e227aae-5428-4eb8-803e-2f6f9b1b94ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:08.960423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e227aae-5428-4eb8-803e-2f6f9b1b94ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:08.961033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e227aae-5428-4eb8-803e-2f6f9b1b94ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:08.962582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e227aae-5428-4eb8-803e-2f6f9b1b94ef admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:08.978252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 796/1586] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:08 2022] POST /identity/v3/users => generated 289 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:08.978443 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 791/1587] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:08 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:08.978560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff08488-cdef-4583-8ff2-a6c593163436 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:08.978674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e227aae-5428-4eb8-803e-2f6f9b1b94ef admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:08.989318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e227aae-5428-4eb8-803e-2f6f9b1b94ef admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:08.994052 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b991b339-bb8d-405c-879f-f10cdd36c82e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:08.994646 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b991b339-bb8d-405c-879f-f10cdd36c82e admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:08.994953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b991b339-bb8d-405c-879f-f10cdd36c82e admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:08.995606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b991b339-bb8d-405c-879f-f10cdd36c82e admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:08.995911 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b991b339-bb8d-405c-879f-f10cdd36c82e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:08.997508 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b991b339-bb8d-405c-879f-f10cdd36c82e admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:09.007681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 792/1588] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:08 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:09.008462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b991b339-bb8d-405c-879f-f10cdd36c82e admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:09.020666 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b991b339-bb8d-405c-879f-f10cdd36c82e admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:09.025314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6010a824-7d42-4fed-ae0b-1802fde7c103 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:09.025911 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6010a824-7d42-4fed-ae0b-1802fde7c103 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:09.026198 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6010a824-7d42-4fed-ae0b-1802fde7c103 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:09.026477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6010a824-7d42-4fed-ae0b-1802fde7c103 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:09.027139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6010a824-7d42-4fed-ae0b-1802fde7c103 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:09.028714 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6010a824-7d42-4fed-ae0b-1802fde7c103 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:09.032658 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff08488-cdef-4583-8ff2-a6c593163436 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:09.036363 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daeeb247-eea3-44f9-b88c-e9d4d4ce6efc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:09.036967 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daeeb247-eea3-44f9-b88c-e9d4d4ce6efc service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:09.037245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daeeb247-eea3-44f9-b88c-e9d4d4ce6efc service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:09.037516 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daeeb247-eea3-44f9-b88c-e9d4d4ce6efc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:09.039826 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 793/1589] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:09 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:09.040798 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6010a824-7d42-4fed-ae0b-1802fde7c103 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:09.046966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daeeb247-eea3-44f9-b88c-e9d4d4ce6efc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:09.048348 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daeeb247-eea3-44f9-b88c-e9d4d4ce6efc service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:09.050290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6010a824-7d42-4fed-ae0b-1802fde7c103 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:09.055852 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d5e7a0-8ed4-4f71-9265-4f410e9a46fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:09.056429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d5e7a0-8ed4-4f71-9265-4f410e9a46fa admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:09.056711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d5e7a0-8ed4-4f71-9265-4f410e9a46fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:09.056983 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d5e7a0-8ed4-4f71-9265-4f410e9a46fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:09.057601 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d5e7a0-8ed4-4f71-9265-4f410e9a46fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:09.057812 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 797/1590] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:09.058577 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daeeb247-eea3-44f9-b88c-e9d4d4ce6efc service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:09.059340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d5e7a0-8ed4-4f71-9265-4f410e9a46fa admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:09.067743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daeeb247-eea3-44f9-b88c-e9d4d4ce6efc service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:09.070851 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 794/1591] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:09.071962 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d5e7a0-8ed4-4f71-9265-4f410e9a46fa admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:09.075610 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0e73c5-30e0-4757-9ffe-6e0dc33f42e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:09.076218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0e73c5-30e0-4757-9ffe-6e0dc33f42e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:09.076505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0e73c5-30e0-4757-9ffe-6e0dc33f42e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:09.077284 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0e73c5-30e0-4757-9ffe-6e0dc33f42e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:09.077921 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c0e73c5-30e0-4757-9ffe-6e0dc33f42e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:09.079555 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c0e73c5-30e0-4757-9ffe-6e0dc33f42e9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:09.081102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=90,keystone:INSERT=10 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:09.084677 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d5e7a0-8ed4-4f71-9265-4f410e9a46fa admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:09.089822 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8acc72-126c-4000-aa8f-5d1e838703c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:09.091170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 798/1592] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:09 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:09.095500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8acc72-126c-4000-aa8f-5d1e838703c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:09.095630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0e73c5-30e0-4757-9ffe-6e0dc33f42e9 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:09.095747 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8acc72-126c-4000-aa8f-5d1e838703c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:09.095747 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8acc72-126c-4000-aa8f-5d1e838703c7 admin admin] PATH_INFO: `/v3/projects/ef5d958ed6b74751b951c2fb90804692/users/e368482ff3834c0fa4a5579e29ae586e/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:09.100162 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0e73c5-30e0-4757-9ffe-6e0dc33f42e9 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:09.105094 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a3589c-b6c1-4e0f-9894-065b455efcf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:09.105926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a3589c-b6c1-4e0f-9894-065b455efcf9 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:09.105926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a3589c-b6c1-4e0f-9894-065b455efcf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:09.106210 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a3589c-b6c1-4e0f-9894-065b455efcf9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:09.106886 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a3589c-b6c1-4e0f-9894-065b455efcf9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:09.108557 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a3589c-b6c1-4e0f-9894-065b455efcf9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:09.108997 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8acc72-126c-4000-aa8f-5d1e838703c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef5d958ed6b74751b951c2fb90804692, user_id=e368482ff3834c0fa4a5579e29ae586e, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:09.110174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: WARNING py.warnings [None req-8f8acc72-126c-4000-aa8f-5d1e838703c7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 03:48:09.110174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: warnings.warn(msg) Sep 16 03:48:09.110174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: Sep 16 03:48:09.112398 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8acc72-126c-4000-aa8f-5d1e838703c7 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:09.119021 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 799/1593] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:09 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:09.119792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a3589c-b6c1-4e0f-9894-065b455efcf9 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:09.129420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a3589c-b6c1-4e0f-9894-065b455efcf9 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:09.133882 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c6771b-cdee-467f-a363-a8ad91ebb111 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:09.134492 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c6771b-cdee-467f-a363-a8ad91ebb111 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:09.134767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c6771b-cdee-467f-a363-a8ad91ebb111 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:09.135065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c6771b-cdee-467f-a363-a8ad91ebb111 admin admin] PATH_INFO: `/v3/projects/ac97d1487c024a68b8ad933d882bb456/users/a6b29b9086f64ea7a7990572567642f7/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:09.149247 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c6771b-cdee-467f-a363-a8ad91ebb111 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac97d1487c024a68b8ad933d882bb456, user_id=a6b29b9086f64ea7a7990572567642f7, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:09.152036 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c6771b-cdee-467f-a363-a8ad91ebb111 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:09.269447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 795/1594] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:09 2022] PUT /identity/v3/projects/ef5d958ed6b74751b951c2fb90804692/users/e368482ff3834c0fa4a5579e29ae586e/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:09.270418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8acc72-126c-4000-aa8f-5d1e838703c7 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:09.299016 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 800/1595] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:09 2022] PUT /identity/v3/projects/ac97d1487c024a68b8ad933d882bb456/users/a6b29b9086f64ea7a7990572567642f7/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:09.303805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c6771b-cdee-467f-a363-a8ad91ebb111 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:09.359843 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8acc72-126c-4000-aa8f-5d1e838703c7 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:09.379425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c6771b-cdee-467f-a363-a8ad91ebb111 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:09.384866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056cd6d0-639b-4811-b0d4-d07ccc8f88be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:09.387594 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056cd6d0-639b-4811-b0d4-d07ccc8f88be admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:09.387594 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056cd6d0-639b-4811-b0d4-d07ccc8f88be admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:09.387594 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056cd6d0-639b-4811-b0d4-d07ccc8f88be admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:09.387594 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056cd6d0-639b-4811-b0d4-d07ccc8f88be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:09.388575 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056cd6d0-639b-4811-b0d4-d07ccc8f88be admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:09.398701 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 801/1596] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:09 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:09.399610 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056cd6d0-639b-4811-b0d4-d07ccc8f88be admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:09.409762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056cd6d0-639b-4811-b0d4-d07ccc8f88be admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:09.414518 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a514ca-36b8-4576-bf19-0503f396a41c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:09.414688 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:09.415416 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a514ca-36b8-4576-bf19-0503f396a41c admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:09.415707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a514ca-36b8-4576-bf19-0503f396a41c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:09.415979 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a514ca-36b8-4576-bf19-0503f396a41c admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:09.416600 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a514ca-36b8-4576-bf19-0503f396a41c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:09.418158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a514ca-36b8-4576-bf19-0503f396a41c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:09.423169 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab0fad8-c7b9-403d-a4ca-05ac4d61cfad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:09.424459 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab0fad8-c7b9-403d-a4ca-05ac4d61cfad admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:09.424869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab0fad8-c7b9-403d-a4ca-05ac4d61cfad admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:09.425253 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab0fad8-c7b9-403d-a4ca-05ac4d61cfad admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:09.426295 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab0fad8-c7b9-403d-a4ca-05ac4d61cfad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:09.427946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab0fad8-c7b9-403d-a4ca-05ac4d61cfad admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:09.428893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 802/1597] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:09.432384 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a514ca-36b8-4576-bf19-0503f396a41c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:09.438769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 796/1598] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:09 2022] GET /identity/v3/roles => generated 1420 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:09.439690 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab0fad8-c7b9-403d-a4ca-05ac4d61cfad admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:09.440313 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a514ca-36b8-4576-bf19-0503f396a41c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:09.447105 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fbb3a5e-b8db-42e6-89b6-272448e4d46f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:09.450764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbb3a5e-b8db-42e6-89b6-272448e4d46f admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:09.450764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbb3a5e-b8db-42e6-89b6-272448e4d46f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:09.450764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbb3a5e-b8db-42e6-89b6-272448e4d46f admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:09.450764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fbb3a5e-b8db-42e6-89b6-272448e4d46f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:09.451184 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab0fad8-c7b9-403d-a4ca-05ac4d61cfad admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:09.451291 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fbb3a5e-b8db-42e6-89b6-272448e4d46f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:09.456029 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41570c9a-66f7-4540-9f1e-17df38987474 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:09.456645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41570c9a-66f7-4540-9f1e-17df38987474 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:09.456930 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41570c9a-66f7-4540-9f1e-17df38987474 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:09.457237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41570c9a-66f7-4540-9f1e-17df38987474 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:09.457876 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41570c9a-66f7-4540-9f1e-17df38987474 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:09.459513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41570c9a-66f7-4540-9f1e-17df38987474 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:09.466066 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 803/1599] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:09 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:09.467882 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5c169a-fa17-4e98-86f9-42c7917fc965 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:09.468502 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5c169a-fa17-4e98-86f9-42c7917fc965 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:09.468783 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5c169a-fa17-4e98-86f9-42c7917fc965 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:09.477133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 797/1600] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:09 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:09.478450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebcfd9a-405d-4ab3-ad8b-e68dae23a9f4 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:09.479482 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebcfd9a-405d-4ab3-ad8b-e68dae23a9f4 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:09.479482 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebcfd9a-405d-4ab3-ad8b-e68dae23a9f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:09.505213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-9ebcfd9a-405d-4ab3-ad8b-e68dae23a9f4 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:09.505352 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9d5c169a-fa17-4e98-86f9-42c7917fc965 None None] MFA Rules not processed for user `e368482ff3834c0fa4a5579e29ae586e`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:09.506485 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-9ebcfd9a-405d-4ab3-ad8b-e68dae23a9f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:09.507206 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9d5c169a-fa17-4e98-86f9-42c7917fc965 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:09.517002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 798/1601] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:09.519760 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebcfd9a-405d-4ab3-ad8b-e68dae23a9f4 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:09.537774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebcfd9a-405d-4ab3-ad8b-e68dae23a9f4 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:09.544347 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbd4c92-0ced-44cf-97bb-c85e067c78b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:09.544907 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbd4c92-0ced-44cf-97bb-c85e067c78b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:09.545197 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbd4c92-0ced-44cf-97bb-c85e067c78b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:09.545525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbd4c92-0ced-44cf-97bb-c85e067c78b1 admin admin] PATH_INFO: `/v3/projects/e94b50976295422c9c8374155cf2d2c2/users/8f410d691e364880b13bb9144bbc5e7d/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:09.565231 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfbd4c92-0ced-44cf-97bb-c85e067c78b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e94b50976295422c9c8374155cf2d2c2, user_id=8f410d691e364880b13bb9144bbc5e7d, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:09.568146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfbd4c92-0ced-44cf-97bb-c85e067c78b1 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:09.588089 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 804/1602] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:09 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:09.588937 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5c169a-fa17-4e98-86f9-42c7917fc965 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:09.603820 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5c169a-fa17-4e98-86f9-42c7917fc965 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:09.607279 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55f0984-ee52-4045-b735-a762faec9ac3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:09.608137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55f0984-ee52-4045-b735-a762faec9ac3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:09.608652 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55f0984-ee52-4045-b735-a762faec9ac3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:09.609058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55f0984-ee52-4045-b735-a762faec9ac3 admin admin] PATH_INFO: `/v3/projects/cb698ddcedae42a29f50203ec59db295/users/416479ba056f4612b6214cf115251269/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:09.626630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55f0984-ee52-4045-b735-a762faec9ac3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb698ddcedae42a29f50203ec59db295, user_id=416479ba056f4612b6214cf115251269, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:09.627963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55f0984-ee52-4045-b735-a762faec9ac3 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:09.650936 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 799/1603] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:09 2022] PUT /identity/v3/projects/e94b50976295422c9c8374155cf2d2c2/users/8f410d691e364880b13bb9144bbc5e7d/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:09.651872 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbd4c92-0ced-44cf-97bb-c85e067c78b1 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:09.709154 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:09.714198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 805/1604] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:09 2022] PUT /identity/v3/projects/cb698ddcedae42a29f50203ec59db295/users/416479ba056f4612b6214cf115251269/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:09.715022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55f0984-ee52-4045-b735-a762faec9ac3 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:09.727325 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbd4c92-0ced-44cf-97bb-c85e067c78b1 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:09.792096 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55f0984-ee52-4045-b735-a762faec9ac3 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:09.798263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e475248d-0818-4b24-81b9-14aded84d454 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:09.799193 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e475248d-0818-4b24-81b9-14aded84d454 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:09.801111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c60f8c-6996-4f1e-9b6a-316259387ee0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:09.801111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c60f8c-6996-4f1e-9b6a-316259387ee0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:09.801111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c60f8c-6996-4f1e-9b6a-316259387ee0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:09.801676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c60f8c-6996-4f1e-9b6a-316259387ee0 admin admin] PATH_INFO: `/v3/projects/d39c13d4012549c2890bf1d904d1c041/users/c1e4846769d44451b239dee5c892b5e8/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:09.802163 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e475248d-0818-4b24-81b9-14aded84d454 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:09.802602 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e475248d-0818-4b24-81b9-14aded84d454 admin admin] PATH_INFO: `/v3/projects/b61c79c5d9714dcf8df92c958dfe5cd0/users/f5784529ef38496a82e2167abff0657b/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:09.816731 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c60f8c-6996-4f1e-9b6a-316259387ee0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d39c13d4012549c2890bf1d904d1c041, user_id=c1e4846769d44451b239dee5c892b5e8, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:09.818237 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e475248d-0818-4b24-81b9-14aded84d454 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b61c79c5d9714dcf8df92c958dfe5cd0, user_id=f5784529ef38496a82e2167abff0657b, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:09.819403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c60f8c-6996-4f1e-9b6a-316259387ee0 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:09.820910 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e475248d-0818-4b24-81b9-14aded84d454 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:09.895056 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 800/1605] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:09 2022] PUT /identity/v3/projects/d39c13d4012549c2890bf1d904d1c041/users/c1e4846769d44451b239dee5c892b5e8/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:09.895271 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 806/1606] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:09 2022] PUT /identity/v3/projects/b61c79c5d9714dcf8df92c958dfe5cd0/users/f5784529ef38496a82e2167abff0657b/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:09.895407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c60f8c-6996-4f1e-9b6a-316259387ee0 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:09.895505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e475248d-0818-4b24-81b9-14aded84d454 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:09.898329 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e475248d-0818-4b24-81b9-14aded84d454 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:09.939981 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:09.961773 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c60f8c-6996-4f1e-9b6a-316259387ee0 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:09.962320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e475248d-0818-4b24-81b9-14aded84d454 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:09.966709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1552da08-f153-4200-a410-277e5b61f5a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:09.968035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1552da08-f153-4200-a410-277e5b61f5a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:09.968935 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1552da08-f153-4200-a410-277e5b61f5a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:09.969269 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1552da08-f153-4200-a410-277e5b61f5a4 admin admin] PATH_INFO: `/v3/projects/e1f32c865ae54bed834184981050369a/users/734b27ebb07948d4839b08ba71b7dc87/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:09.985579 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1552da08-f153-4200-a410-277e5b61f5a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1f32c865ae54bed834184981050369a, user_id=734b27ebb07948d4839b08ba71b7dc87, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:09.988742 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1552da08-f153-4200-a410-277e5b61f5a4 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:10.019812 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77fe181-dd03-4eff-ab18-99eab95aca79 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:10.020430 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77fe181-dd03-4eff-ab18-99eab95aca79 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:10.020705 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77fe181-dd03-4eff-ab18-99eab95aca79 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:10.020982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77fe181-dd03-4eff-ab18-99eab95aca79 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:10.030396 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77fe181-dd03-4eff-ab18-99eab95aca79 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:10.031858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77fe181-dd03-4eff-ab18-99eab95aca79 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:10.041431 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 807/1607] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:10.042664 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afaff6f-16db-4b5f-8e6e-5d0a82fbd3d8 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:10.042969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afaff6f-16db-4b5f-8e6e-5d0a82fbd3d8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:10.043260 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afaff6f-16db-4b5f-8e6e-5d0a82fbd3d8 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:10.044448 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 808/1608] 10.222.0.49 () {54 vars in 962 bytes} [Fri Sep 16 03:48:10 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:48:10.045461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f095e5a-3453-45e2-8f11-56467edae549 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:10.045750 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f095e5a-3453-45e2-8f11-56467edae549 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:10.046036 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f095e5a-3453-45e2-8f11-56467edae549 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:10.063086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 801/1609] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:09 2022] PUT /identity/v3/projects/e1f32c865ae54bed834184981050369a/users/734b27ebb07948d4839b08ba71b7dc87/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:10.064361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f9e6f7-254a-413a-9e43-bde0be8f3cb8 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:10.064646 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f9e6f7-254a-413a-9e43-bde0be8f3cb8 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:10.064919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f9e6f7-254a-413a-9e43-bde0be8f3cb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:10.067827 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9f095e5a-3453-45e2-8f11-56467edae549 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:10.069224 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9f095e5a-3453-45e2-8f11-56467edae549 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:10.091310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-a1f9e6f7-254a-413a-9e43-bde0be8f3cb8 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:10.092722 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a1f9e6f7-254a-413a-9e43-bde0be8f3cb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:10.115005 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:10.137515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 809/1610] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:10.138768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af83c248-4d78-46f9-bc27-f86944fcb443 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:10.139091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af83c248-4d78-46f9-bc27-f86944fcb443 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:10.140057 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af83c248-4d78-46f9-bc27-f86944fcb443 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:10.152913 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 802/1611] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:10.157517 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195dcc1f-9aa9-450c-bb89-9a61e8c0447c None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:10.157517 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195dcc1f-9aa9-450c-bb89-9a61e8c0447c None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:10.157517 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195dcc1f-9aa9-450c-bb89-9a61e8c0447c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:10.159482 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-af83c248-4d78-46f9-bc27-f86944fcb443 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:10.160702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-af83c248-4d78-46f9-bc27-f86944fcb443 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:10.168750 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 810/1612] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:10.173857 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e673382-e004-4db6-aebf-1fad1a5068e7 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:10.175165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e673382-e004-4db6-aebf-1fad1a5068e7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:10.176038 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e673382-e004-4db6-aebf-1fad1a5068e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:10.194298 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-195dcc1f-9aa9-450c-bb89-9a61e8c0447c None None] MFA Rules not processed for user `f5784529ef38496a82e2167abff0657b`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:10.195559 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-195dcc1f-9aa9-450c-bb89-9a61e8c0447c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:10.217856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8e673382-e004-4db6-aebf-1fad1a5068e7 None None] MFA Rules not processed for user `ded1bcd10e62499a9aaa0ece5aa55c80`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:10.219221 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8e673382-e004-4db6-aebf-1fad1a5068e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:10.277582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 803/1613] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:10 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:10.277582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4728fdde-bca3-47a8-9cf9-387532450e90 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:10.277582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4728fdde-bca3-47a8-9cf9-387532450e90 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:10.277582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4728fdde-bca3-47a8-9cf9-387532450e90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:10.308968 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 811/1614] 10.222.0.49 () {58 vars in 1059 bytes} [Fri Sep 16 03:48:10 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:10.310068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e673382-e004-4db6-aebf-1fad1a5068e7 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:10.328102 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-4728fdde-bca3-47a8-9cf9-387532450e90 None None] MFA Rules not processed for user `734b27ebb07948d4839b08ba71b7dc87`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:10.329243 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-4728fdde-bca3-47a8-9cf9-387532450e90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:10.380183 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e673382-e004-4db6-aebf-1fad1a5068e7 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:10.384403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:10.384996 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d174d8-62af-4cce-a0e6-8f246209fc4f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:10.385618 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d174d8-62af-4cce-a0e6-8f246209fc4f service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:10.385902 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d174d8-62af-4cce-a0e6-8f246209fc4f service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:10.386403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d174d8-62af-4cce-a0e6-8f246209fc4f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:10.396250 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d174d8-62af-4cce-a0e6-8f246209fc4f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:10.397678 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d174d8-62af-4cce-a0e6-8f246209fc4f service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:10.407970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 812/1615] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:10.412013 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 804/1616] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:10 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:10.412134 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d174d8-62af-4cce-a0e6-8f246209fc4f service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:10.412254 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4728fdde-bca3-47a8-9cf9-387532450e90 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:10.419256 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4728fdde-bca3-47a8-9cf9-387532450e90 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:10.420085 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d174d8-62af-4cce-a0e6-8f246209fc4f service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:10.424711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ae07fd-ff31-4426-90cc-bd3965f62ecf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:10.425765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ae07fd-ff31-4426-90cc-bd3965f62ecf service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:10.425894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fb7453-cd2f-4cd5-8458-2b171903f6f2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:10.425993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ae07fd-ff31-4426-90cc-bd3965f62ecf service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:10.426322 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ae07fd-ff31-4426-90cc-bd3965f62ecf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:10.426705 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fb7453-cd2f-4cd5-8458-2b171903f6f2 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:10.426826 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fb7453-cd2f-4cd5-8458-2b171903f6f2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:10.427350 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fb7453-cd2f-4cd5-8458-2b171903f6f2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:10.437884 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ae07fd-ff31-4426-90cc-bd3965f62ecf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:10.446586 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fb7453-cd2f-4cd5-8458-2b171903f6f2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:10.446739 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ae07fd-ff31-4426-90cc-bd3965f62ecf service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:10.446823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fb7453-cd2f-4cd5-8458-2b171903f6f2 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:10.449816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 805/1617] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:10.453027 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 813/1618] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:10.455870 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fb7453-cd2f-4cd5-8458-2b171903f6f2 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:10.457106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ae07fd-ff31-4426-90cc-bd3965f62ecf service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:10.469376 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fb7453-cd2f-4cd5-8458-2b171903f6f2 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:10.475064 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ae07fd-ff31-4426-90cc-bd3965f62ecf service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:10.476630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927ad14f-914f-43f8-808a-7884b451094a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:10.476889 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927ad14f-914f-43f8-808a-7884b451094a service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:10.477069 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aea58f0-02e3-40f3-9ceb-7f39ed7de009 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:10.477212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927ad14f-914f-43f8-808a-7884b451094a service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:10.477433 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927ad14f-914f-43f8-808a-7884b451094a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:10.477767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aea58f0-02e3-40f3-9ceb-7f39ed7de009 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:10.478169 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aea58f0-02e3-40f3-9ceb-7f39ed7de009 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:10.478655 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aea58f0-02e3-40f3-9ceb-7f39ed7de009 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:10.479471 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aea58f0-02e3-40f3-9ceb-7f39ed7de009 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:10.482464 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aea58f0-02e3-40f3-9ceb-7f39ed7de009 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:10.527685 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:10.543977 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927ad14f-914f-43f8-808a-7884b451094a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:10.546424 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927ad14f-914f-43f8-808a-7884b451094a service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:10.561242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 814/1619] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:10 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:10.561242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927ad14f-914f-43f8-808a-7884b451094a service nova] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:10.569093 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927ad14f-914f-43f8-808a-7884b451094a service nova] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:10.574029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6231734c-2f80-47ef-91a1-1b78c067cb95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:10.574627 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6231734c-2f80-47ef-91a1-1b78c067cb95 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:10.574922 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6231734c-2f80-47ef-91a1-1b78c067cb95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:10.575406 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6231734c-2f80-47ef-91a1-1b78c067cb95 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:10.576054 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6231734c-2f80-47ef-91a1-1b78c067cb95 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:10.584029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6231734c-2f80-47ef-91a1-1b78c067cb95 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:10.619879 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 806/1620] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:10 2022] POST /identity/v3/projects => generated 401 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:10.628614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aea58f0-02e3-40f3-9ceb-7f39ed7de009 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:10.700740 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 815/1621] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:10 2022] POST /identity/v3/projects => generated 379 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:10.705080 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6231734c-2f80-47ef-91a1-1b78c067cb95 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:10.793567 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aea58f0-02e3-40f3-9ceb-7f39ed7de009 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:10.810612 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc6ac6d-7adc-4921-bf7f-64cbcfd0e875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:10.811904 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc6ac6d-7adc-4921-bf7f-64cbcfd0e875 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:10.812136 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc6ac6d-7adc-4921-bf7f-64cbcfd0e875 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:10.812439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc6ac6d-7adc-4921-bf7f-64cbcfd0e875 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:10.813132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc6ac6d-7adc-4921-bf7f-64cbcfd0e875 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:10.815026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc6ac6d-7adc-4921-bf7f-64cbcfd0e875 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:10.820719 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6231734c-2f80-47ef-91a1-1b78c067cb95 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:10.832626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf69f5ea-e0a5-45c0-9fe4-e3eb905d44b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:10.833656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf69f5ea-e0a5-45c0-9fe4-e3eb905d44b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:10.833763 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf69f5ea-e0a5-45c0-9fe4-e3eb905d44b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:10.834085 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf69f5ea-e0a5-45c0-9fe4-e3eb905d44b5 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:10.834647 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf69f5ea-e0a5-45c0-9fe4-e3eb905d44b5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:10.838354 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf69f5ea-e0a5-45c0-9fe4-e3eb905d44b5 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:10.983896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 816/1622] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:10 2022] POST /identity/v3/users => generated 290 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:10.989503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603745ba-1c49-47f4-8690-0c091df4a5c6 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:10.989503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603745ba-1c49-47f4-8690-0c091df4a5c6 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:10.989503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603745ba-1c49-47f4-8690-0c091df4a5c6 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:10.989503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 817/1623] 10.222.0.49 () {54 vars in 912 bytes} [Fri Sep 16 03:48:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:48:10.989503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603745ba-1c49-47f4-8690-0c091df4a5c6 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:11.005330 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603745ba-1c49-47f4-8690-0c091df4a5c6 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:11.010948 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30339f08-5acf-4b2c-9439-bab2006e3a2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:11.012244 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30339f08-5acf-4b2c-9439-bab2006e3a2a admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:11.012882 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30339f08-5acf-4b2c-9439-bab2006e3a2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:11.013373 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30339f08-5acf-4b2c-9439-bab2006e3a2a admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:11.014204 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30339f08-5acf-4b2c-9439-bab2006e3a2a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:11.016365 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30339f08-5acf-4b2c-9439-bab2006e3a2a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:11.021064 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 807/1624] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:10 2022] POST /identity/v3/users => generated 307 bytes in 393 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:11.022543 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4936bb65-087a-4c83-a388-036de6b5e1f5 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:11.022842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4936bb65-087a-4c83-a388-036de6b5e1f5 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:11.023350 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4936bb65-087a-4c83-a388-036de6b5e1f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:11.034764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 818/1625] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:10 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:11.034764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30339f08-5acf-4b2c-9439-bab2006e3a2a admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:11.048881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30339f08-5acf-4b2c-9439-bab2006e3a2a admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:11.058660 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-4936bb65-087a-4c83-a388-036de6b5e1f5 None None] MFA Rules not processed for user `ded1bcd10e62499a9aaa0ece5aa55c80`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:11.058828 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9bdb13-2cb3-4908-b33c-29dcab095c6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:11.058828 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9bdb13-2cb3-4908-b33c-29dcab095c6d admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:11.059079 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-4936bb65-087a-4c83-a388-036de6b5e1f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:11.059172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9bdb13-2cb3-4908-b33c-29dcab095c6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:11.059172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9bdb13-2cb3-4908-b33c-29dcab095c6d admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:11.059473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9bdb13-2cb3-4908-b33c-29dcab095c6d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:11.062190 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9bdb13-2cb3-4908-b33c-29dcab095c6d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:11.078135 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 819/1626] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:11 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:11.079232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9bdb13-2cb3-4908-b33c-29dcab095c6d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:11.110802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9bdb13-2cb3-4908-b33c-29dcab095c6d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:11.130130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4797c4-9f9b-46bd-ae6b-ddd79b31f225 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:11.130777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4797c4-9f9b-46bd-ae6b-ddd79b31f225 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:11.131085 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4797c4-9f9b-46bd-ae6b-ddd79b31f225 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:11.131358 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4797c4-9f9b-46bd-ae6b-ddd79b31f225 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:11.132239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4797c4-9f9b-46bd-ae6b-ddd79b31f225 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:11.133850 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4797c4-9f9b-46bd-ae6b-ddd79b31f225 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:11.160064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 820/1627] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:11 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:11.160994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4797c4-9f9b-46bd-ae6b-ddd79b31f225 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:11.181060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4797c4-9f9b-46bd-ae6b-ddd79b31f225 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:11.187379 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9c2ef0-3181-447e-8b98-fc111ec0d118 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:11.188048 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9c2ef0-3181-447e-8b98-fc111ec0d118 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:11.188326 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9c2ef0-3181-447e-8b98-fc111ec0d118 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:11.188594 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9c2ef0-3181-447e-8b98-fc111ec0d118 admin admin] PATH_INFO: `/v3/projects/4e9e0e53ce3f4f2bbcc0227e9d118db0/users/d27f26d6061f41848f52cd6d4416a97f/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:11.201026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 808/1628] 10.222.0.49 () {58 vars in 1009 bytes} [Fri Sep 16 03:48:11 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:11.202260 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4936bb65-087a-4c83-a388-036de6b5e1f5 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:11.224744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9c2ef0-3181-447e-8b98-fc111ec0d118 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4e9e0e53ce3f4f2bbcc0227e9d118db0, user_id=d27f26d6061f41848f52cd6d4416a97f, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:11.231108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4936bb65-087a-4c83-a388-036de6b5e1f5 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:11.233257 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9c2ef0-3181-447e-8b98-fc111ec0d118 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:11.241747 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3364e8f-477c-465d-8da7-6bdcafb5de20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:11.242484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3364e8f-477c-465d-8da7-6bdcafb5de20 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:11.242773 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3364e8f-477c-465d-8da7-6bdcafb5de20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:11.243092 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3364e8f-477c-465d-8da7-6bdcafb5de20 admin admin] PATH_INFO: `/v3/projects/64845a46eab34d8bbbc9a4606c1a7b35/users/c4679421808b42cdb68b1a02a2a0675b/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:11.263393 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3364e8f-477c-465d-8da7-6bdcafb5de20 admin admin] RBAC: Authorizing `identity:create_grant(project_id=64845a46eab34d8bbbc9a4606c1a7b35, user_id=c4679421808b42cdb68b1a02a2a0675b, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:11.266260 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3364e8f-477c-465d-8da7-6bdcafb5de20 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:11.371983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 821/1629] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:11 2022] PUT /identity/v3/projects/4e9e0e53ce3f4f2bbcc0227e9d118db0/users/d27f26d6061f41848f52cd6d4416a97f/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:11.371983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9c2ef0-3181-447e-8b98-fc111ec0d118 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:11.452616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 809/1630] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:11 2022] PUT /identity/v3/projects/64845a46eab34d8bbbc9a4606c1a7b35/users/c4679421808b42cdb68b1a02a2a0675b/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:11.453990 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3364e8f-477c-465d-8da7-6bdcafb5de20 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:11.467895 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:11.474323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9c2ef0-3181-447e-8b98-fc111ec0d118 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:11.488507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:11.541094 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e327752e-7f78-49e7-9d01-4f0594cdf3bb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:11.542134 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e327752e-7f78-49e7-9d01-4f0594cdf3bb service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:11.542689 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e327752e-7f78-49e7-9d01-4f0594cdf3bb service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:11.543183 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e327752e-7f78-49e7-9d01-4f0594cdf3bb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:11.975209 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3364e8f-477c-465d-8da7-6bdcafb5de20 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:11.975209 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71033d2c-7674-4038-905b-292c4bf5d472 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:11.975209 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71033d2c-7674-4038-905b-292c4bf5d472 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:11.975209 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71033d2c-7674-4038-905b-292c4bf5d472 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:11.978403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71033d2c-7674-4038-905b-292c4bf5d472 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:11.978403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71033d2c-7674-4038-905b-292c4bf5d472 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:11.978403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71033d2c-7674-4038-905b-292c4bf5d472 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:11.978403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 810/1631] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:11 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:11.978403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c192934-1852-4259-a6ad-904d42bc84b7 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:11.978403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c192934-1852-4259-a6ad-904d42bc84b7 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:11.979116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c192934-1852-4259-a6ad-904d42bc84b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:11.979116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-0c192934-1852-4259-a6ad-904d42bc84b7 None None] MFA Rules not processed for user `c4679421808b42cdb68b1a02a2a0675b`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:11.979116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-0c192934-1852-4259-a6ad-904d42bc84b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:11.979762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e327752e-7f78-49e7-9d01-4f0594cdf3bb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:11.979762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e327752e-7f78-49e7-9d01-4f0594cdf3bb service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:11.980836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 811/1632] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:11 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:11.980836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c192934-1852-4259-a6ad-904d42bc84b7 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:11.980836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c192934-1852-4259-a6ad-904d42bc84b7 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:11.980836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340daf90-69f7-4777-b9ee-aafb7981e6ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:11.981296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340daf90-69f7-4777-b9ee-aafb7981e6ae admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:11.981296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340daf90-69f7-4777-b9ee-aafb7981e6ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:11.981296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340daf90-69f7-4777-b9ee-aafb7981e6ae admin admin] PATH_INFO: `/v3/projects/4e9e0e53ce3f4f2bbcc0227e9d118db0/users/d27f26d6061f41848f52cd6d4416a97f/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:11.981296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340daf90-69f7-4777-b9ee-aafb7981e6ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=4e9e0e53ce3f4f2bbcc0227e9d118db0, user_id=d27f26d6061f41848f52cd6d4416a97f, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:11.981296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340daf90-69f7-4777-b9ee-aafb7981e6ae admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:11.981808 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 822/1633] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:11 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:11.981808 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e327752e-7f78-49e7-9d01-4f0594cdf3bb service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:11.981808 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e327752e-7f78-49e7-9d01-4f0594cdf3bb service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:11.983816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 812/1634] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:11 2022] PUT /identity/v3/projects/4e9e0e53ce3f4f2bbcc0227e9d118db0/users/d27f26d6061f41848f52cd6d4416a97f/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:11.983816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7780e92-d36e-4a79-a497-40ac4f809043 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:11.983816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7780e92-d36e-4a79-a497-40ac4f809043 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:11.983816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7780e92-d36e-4a79-a497-40ac4f809043 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:11.988041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234c6f28-77bc-4fd9-aa57-cce3e2fbb3bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:11.988041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234c6f28-77bc-4fd9-aa57-cce3e2fbb3bd admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:11.988041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234c6f28-77bc-4fd9-aa57-cce3e2fbb3bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:11.988041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234c6f28-77bc-4fd9-aa57-cce3e2fbb3bd admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:11.988369 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234c6f28-77bc-4fd9-aa57-cce3e2fbb3bd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:11.988369 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234c6f28-77bc-4fd9-aa57-cce3e2fbb3bd admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:11.989125 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-d7780e92-d36e-4a79-a497-40ac4f809043 None None] MFA Rules not processed for user `d27f26d6061f41848f52cd6d4416a97f`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:11.989125 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d7780e92-d36e-4a79-a497-40ac4f809043 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:12.048115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 813/1635] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:11 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:48:12.060988 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7780e92-d36e-4a79-a497-40ac4f809043 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:12.135191 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 823/1636] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:11 2022] POST /identity/v3/projects => generated 379 bytes in 340 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:12.138462 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234c6f28-77bc-4fd9-aa57-cce3e2fbb3bd admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:12.182127 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7780e92-d36e-4a79-a497-40ac4f809043 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:12.301870 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d08a653-9f9a-4a4c-9012-e84fe5fb2881 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:12.301870 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d08a653-9f9a-4a4c-9012-e84fe5fb2881 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:12.301870 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d08a653-9f9a-4a4c-9012-e84fe5fb2881 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:12.302128 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234c6f28-77bc-4fd9-aa57-cce3e2fbb3bd admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:12.302225 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d08a653-9f9a-4a4c-9012-e84fe5fb2881 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:12.305507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9d0f86-f98b-4d5e-be18-091cd138a66f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:12.306420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9d0f86-f98b-4d5e-be18-091cd138a66f admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:12.306420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9d0f86-f98b-4d5e-be18-091cd138a66f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:12.306681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9d0f86-f98b-4d5e-be18-091cd138a66f admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:12.308733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9d0f86-f98b-4d5e-be18-091cd138a66f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:12.312562 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9d0f86-f98b-4d5e-be18-091cd138a66f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:12.413263 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d08a653-9f9a-4a4c-9012-e84fe5fb2881 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:12.417517 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d08a653-9f9a-4a4c-9012-e84fe5fb2881 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:12.439706 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 814/1637] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:12 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:48:12.448555 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 824/1638] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:12 2022] POST /identity/v3/users => generated 296 bytes in 310 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:12.449624 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d08a653-9f9a-4a4c-9012-e84fe5fb2881 service cinder] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:12.471709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d08a653-9f9a-4a4c-9012-e84fe5fb2881 service cinder] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:12.477420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e4935a-fb14-4843-a727-a326493f392a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:12.482603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e4935a-fb14-4843-a727-a326493f392a admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:12.482603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e4935a-fb14-4843-a727-a326493f392a admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:12.482603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e4935a-fb14-4843-a727-a326493f392a admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:12.482603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e4935a-fb14-4843-a727-a326493f392a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:12.482603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e4935a-fb14-4843-a727-a326493f392a admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:12.500479 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9d0f86-f98b-4d5e-be18-091cd138a66f admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:12.506192 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 815/1639] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:12 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:12.511913 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9d0f86-f98b-4d5e-be18-091cd138a66f admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:12.519963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9a39a1-7c7f-4925-945f-00b0cc4ecfc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:12.520654 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9a39a1-7c7f-4925-945f-00b0cc4ecfc0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:12.520923 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9a39a1-7c7f-4925-945f-00b0cc4ecfc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:12.521201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9a39a1-7c7f-4925-945f-00b0cc4ecfc0 admin admin] PATH_INFO: `/v3/projects/655dca434c2647bcb0e935bfb1dc9b42/users/c2f718ebaee24d98a51df42a33009eff/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:12.577509 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9a39a1-7c7f-4925-945f-00b0cc4ecfc0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=655dca434c2647bcb0e935bfb1dc9b42, user_id=c2f718ebaee24d98a51df42a33009eff, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:12.580944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9a39a1-7c7f-4925-945f-00b0cc4ecfc0 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:12.714271 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 825/1640] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:12 2022] PUT /identity/v3/projects/655dca434c2647bcb0e935bfb1dc9b42/users/c2f718ebaee24d98a51df42a33009eff/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:48:12.719098 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e4935a-fb14-4843-a727-a326493f392a admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:12.738122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e4935a-fb14-4843-a727-a326493f392a admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:12.782463 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9a39a1-7c7f-4925-945f-00b0cc4ecfc0 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:12.837795 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6976fa5a-d009-4a1f-8511-42882e8281ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:12.838676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6976fa5a-d009-4a1f-8511-42882e8281ac admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:12.838812 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6976fa5a-d009-4a1f-8511-42882e8281ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:12.839491 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6976fa5a-d009-4a1f-8511-42882e8281ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:12.840174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6976fa5a-d009-4a1f-8511-42882e8281ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:12.841872 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6976fa5a-d009-4a1f-8511-42882e8281ac admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:12.854945 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 816/1641] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:12 2022] GET /identity/v3/roles => generated 1420 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:12.861599 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6976fa5a-d009-4a1f-8511-42882e8281ac admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:12.874343 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6976fa5a-d009-4a1f-8511-42882e8281ac admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:12.876336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9a39a1-7c7f-4925-945f-00b0cc4ecfc0 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:12.882034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f83cdef-5ce5-4f29-a616-4f35ede91aa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:12.882683 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730d0ab9-6cae-490e-93e5-ef79b3db66ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:12.883221 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f83cdef-5ce5-4f29-a616-4f35ede91aa6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:12.883384 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730d0ab9-6cae-490e-93e5-ef79b3db66ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:12.883729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730d0ab9-6cae-490e-93e5-ef79b3db66ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:12.883954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730d0ab9-6cae-490e-93e5-ef79b3db66ee admin admin] PATH_INFO: `/v3/users/f5784529ef38496a82e2167abff0657b` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:12.884370 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f83cdef-5ce5-4f29-a616-4f35ede91aa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:12.884790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f83cdef-5ce5-4f29-a616-4f35ede91aa6 admin admin] PATH_INFO: `/v3/projects/655dca434c2647bcb0e935bfb1dc9b42/users/c2f718ebaee24d98a51df42a33009eff/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:12.885291 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730d0ab9-6cae-490e-93e5-ef79b3db66ee admin admin] RBAC: Authorizing `identity:delete_user(user_id=f5784529ef38496a82e2167abff0657b)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:12.887183 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f83cdef-5ce5-4f29-a616-4f35ede91aa6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=655dca434c2647bcb0e935bfb1dc9b42, user_id=c2f718ebaee24d98a51df42a33009eff, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:12.887605 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730d0ab9-6cae-490e-93e5-ef79b3db66ee admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:12.891697 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f83cdef-5ce5-4f29-a616-4f35ede91aa6 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:12.916516 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 817/1642] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:12 2022] PUT /identity/v3/projects/655dca434c2647bcb0e935bfb1dc9b42/users/c2f718ebaee24d98a51df42a33009eff/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:12.928140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc46b15b-43fa-40e2-bbd1-f2ea8bdf6119 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:12.928481 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc46b15b-43fa-40e2-bbd1-f2ea8bdf6119 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:12.928846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc46b15b-43fa-40e2-bbd1-f2ea8bdf6119 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:12.972595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-cc46b15b-43fa-40e2-bbd1-f2ea8bdf6119 None None] MFA Rules not processed for user `c2f718ebaee24d98a51df42a33009eff`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:12.973907 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-cc46b15b-43fa-40e2-bbd1-f2ea8bdf6119 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:12.979078 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:13.049501 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 818/1643] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:12 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:48:13.061651 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc46b15b-43fa-40e2-bbd1-f2ea8bdf6119 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:13.074002 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:13.159849 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc46b15b-43fa-40e2-bbd1-f2ea8bdf6119 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:13.164668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc52bdd7-cd4c-4e48-8b1a-bbb3b919b8f8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:13.165425 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc52bdd7-cd4c-4e48-8b1a-bbb3b919b8f8 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:13.165770 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc52bdd7-cd4c-4e48-8b1a-bbb3b919b8f8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:13.166150 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc52bdd7-cd4c-4e48-8b1a-bbb3b919b8f8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:13.183465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc52bdd7-cd4c-4e48-8b1a-bbb3b919b8f8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:13.185056 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc52bdd7-cd4c-4e48-8b1a-bbb3b919b8f8 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:13.200959 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 819/1644] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:13 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:48:13.237113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-730d0ab9-6cae-490e-93e5-ef79b3db66ee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f5784529ef38496a82e2167abff0657b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:13.261602 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-730d0ab9-6cae-490e-93e5-ef79b3db66ee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f5784529ef38496a82e2167abff0657b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:13.411611 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-730d0ab9-6cae-490e-93e5-ef79b3db66ee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f5784529ef38496a82e2167abff0657b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:13.412873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-730d0ab9-6cae-490e-93e5-ef79b3db66ee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f5784529ef38496a82e2167abff0657b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:13.430512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-730d0ab9-6cae-490e-93e5-ef79b3db66ee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f5784529ef38496a82e2167abff0657b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:13.440483 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 826/1645] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:48:12 2022] DELETE /identity/v3/users/f5784529ef38496a82e2167abff0657b => generated 0 bytes in 659 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:13.444815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2caac98-31c6-4ee0-ac71-2fa997c2eb6c None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:13.445100 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2caac98-31c6-4ee0-ac71-2fa997c2eb6c None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:13.445376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2caac98-31c6-4ee0-ac71-2fa997c2eb6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:13.446079 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730d0ab9-6cae-490e-93e5-ef79b3db66ee admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:13.449755 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-730d0ab9-6cae-490e-93e5-ef79b3db66ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:13.480952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-c2caac98-31c6-4ee0-ac71-2fa997c2eb6c None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:13.482215 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c2caac98-31c6-4ee0-ac71-2fa997c2eb6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:13.601613 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730d0ab9-6cae-490e-93e5-ef79b3db66ee admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:13.615643 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c66d06f-69a5-4645-abd9-f73ef1f95ca0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:13.617925 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c66d06f-69a5-4645-abd9-f73ef1f95ca0 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:13.618276 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c66d06f-69a5-4645-abd9-f73ef1f95ca0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:13.618567 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c66d06f-69a5-4645-abd9-f73ef1f95ca0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:13.627844 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5c66d06f-69a5-4645-abd9-f73ef1f95ca0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:13.656003 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 820/1646] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:13.677016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2caac98-31c6-4ee0-ac71-2fa997c2eb6c None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:13.682185 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c2caac98-31c6-4ee0-ac71-2fa997c2eb6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:13.815706 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c66d06f-69a5-4645-abd9-f73ef1f95ca0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:13.820903 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c66d06f-69a5-4645-abd9-f73ef1f95ca0 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:13.830252 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2caac98-31c6-4ee0-ac71-2fa997c2eb6c None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:13.836375 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd69e72d-4d21-476a-bb1d-c66f66dea39f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:13.840656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd69e72d-4d21-476a-bb1d-c66f66dea39f service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:13.840656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd69e72d-4d21-476a-bb1d-c66f66dea39f service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:13.840656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd69e72d-4d21-476a-bb1d-c66f66dea39f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:13.844811 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 827/1647] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:13 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:13.854995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd69e72d-4d21-476a-bb1d-c66f66dea39f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:13.854995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd69e72d-4d21-476a-bb1d-c66f66dea39f service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:13.868288 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 821/1648] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:13.983920 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2075e755-1cf2-41f1-a2b0-02dc5e8136dd None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:13.989747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2075e755-1cf2-41f1-a2b0-02dc5e8136dd None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:13.989747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2075e755-1cf2-41f1-a2b0-02dc5e8136dd None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:13.989747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 828/1649] 10.222.0.49 () {54 vars in 962 bytes} [Fri Sep 16 03:48:13 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:48:14.000502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c888f4-de38-4411-8eff-97a4d214af5f None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:14.000943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c888f4-de38-4411-8eff-97a4d214af5f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:14.001127 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c888f4-de38-4411-8eff-97a4d214af5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:14.041135 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-70c888f4-de38-4411-8eff-97a4d214af5f None None] MFA Rules not processed for user `ded1bcd10e62499a9aaa0ece5aa55c80`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:14.042459 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-70c888f4-de38-4411-8eff-97a4d214af5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:14.169091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 822/1650] 10.222.0.49 () {58 vars in 1059 bytes} [Fri Sep 16 03:48:13 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:14.175146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7005f33-d6c2-415d-8ff7-c835d0b8ce78 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:14.175988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7005f33-d6c2-415d-8ff7-c835d0b8ce78 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:14.176183 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7005f33-d6c2-415d-8ff7-c835d0b8ce78 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:14.177507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 829/1651] 10.222.0.49 () {54 vars in 912 bytes} [Fri Sep 16 03:48:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:48:14.188902 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1841552b-7313-46b9-bf85-8678f8622a81 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:14.190240 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1841552b-7313-46b9-bf85-8678f8622a81 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:14.190533 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1841552b-7313-46b9-bf85-8678f8622a81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:14.229532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-1841552b-7313-46b9-bf85-8678f8622a81 None None] MFA Rules not processed for user `ded1bcd10e62499a9aaa0ece5aa55c80`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:14.230754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1841552b-7313-46b9-bf85-8678f8622a81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:14.242895 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 823/1652] 10.222.0.49 () {58 vars in 1009 bytes} [Fri Sep 16 03:48:14 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:14.256597 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7005f33-d6c2-415d-8ff7-c835d0b8ce78 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:14.259188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b7005f33-d6c2-415d-8ff7-c835d0b8ce78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:14.274381 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7005f33-d6c2-415d-8ff7-c835d0b8ce78 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:14.278951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178dd2db-fca4-47a0-8cb7-1546a035e0f2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:14.280152 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178dd2db-fca4-47a0-8cb7-1546a035e0f2 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:14.284494 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178dd2db-fca4-47a0-8cb7-1546a035e0f2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:14.284494 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178dd2db-fca4-47a0-8cb7-1546a035e0f2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:14.292873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178dd2db-fca4-47a0-8cb7-1546a035e0f2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:14.294605 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178dd2db-fca4-47a0-8cb7-1546a035e0f2 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:14.328070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 830/1653] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:14 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:14.823147 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:14.998052 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1841552b-7313-46b9-bf85-8678f8622a81 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:15.000975 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1841552b-7313-46b9-bf85-8678f8622a81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:15.037058 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1841552b-7313-46b9-bf85-8678f8622a81 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:15.052103 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4b2c77-d8f3-40e0-8dab-4ab5a5e40efb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:15.062309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4b2c77-d8f3-40e0-8dab-4ab5a5e40efb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:15.069186 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4b2c77-d8f3-40e0-8dab-4ab5a5e40efb admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:15.069804 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4b2c77-d8f3-40e0-8dab-4ab5a5e40efb admin admin] PATH_INFO: `/v3/projects/b61c79c5d9714dcf8df92c958dfe5cd0` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:15.072663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4b2c77-d8f3-40e0-8dab-4ab5a5e40efb admin admin] RBAC: Authorizing `identity:delete_project(project_id=b61c79c5d9714dcf8df92c958dfe5cd0)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:15.082498 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4b2c77-d8f3-40e0-8dab-4ab5a5e40efb admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:15.142927 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=85,keystone:INSERT=5,keystone:DELETE=10 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:15.339658 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-3c4b2c77-d8f3-40e0-8dab-4ab5a5e40efb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b61c79c5d9714dcf8df92c958dfe5cd0', 'request_id': 'req-3c4b2c77-d8f3-40e0-8dab-4ab5a5e40efb'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:15.491647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-3c4b2c77-d8f3-40e0-8dab-4ab5a5e40efb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b61c79c5d9714dcf8df92c958dfe5cd0', 'request_id': 'req-3c4b2c77-d8f3-40e0-8dab-4ab5a5e40efb'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:15.501634 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-3c4b2c77-d8f3-40e0-8dab-4ab5a5e40efb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b61c79c5d9714dcf8df92c958dfe5cd0', 'request_id': 'req-3c4b2c77-d8f3-40e0-8dab-4ab5a5e40efb'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:15.543773 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-3c4b2c77-d8f3-40e0-8dab-4ab5a5e40efb admin admin] The token cache is being invalidate because project b61c79c5d9714dcf8df92c958dfe5cd0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:48:15.544337 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-3c4b2c77-d8f3-40e0-8dab-4ab5a5e40efb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:15.549229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-3c4b2c77-d8f3-40e0-8dab-4ab5a5e40efb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:15.557645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 824/1654] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:48:14 2022] DELETE /identity/v3/projects/b61c79c5d9714dcf8df92c958dfe5cd0 => generated 0 bytes in 561 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:15.563525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178dd2db-fca4-47a0-8cb7-1546a035e0f2 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:15.579747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178dd2db-fca4-47a0-8cb7-1546a035e0f2 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:15.680604 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278e764e-49e8-49ce-bd77-aaebaece18aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:15.681436 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278e764e-49e8-49ce-bd77-aaebaece18aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:15.681813 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278e764e-49e8-49ce-bd77-aaebaece18aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:15.682185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278e764e-49e8-49ce-bd77-aaebaece18aa admin admin] PATH_INFO: `/v3/users/d27f26d6061f41848f52cd6d4416a97f` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:15.683582 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278e764e-49e8-49ce-bd77-aaebaece18aa admin admin] RBAC: Authorizing `identity:delete_user(user_id=d27f26d6061f41848f52cd6d4416a97f)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:15.685870 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278e764e-49e8-49ce-bd77-aaebaece18aa admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:16.066313 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-278e764e-49e8-49ce-bd77-aaebaece18aa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd27f26d6061f41848f52cd6d4416a97f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:16.112380 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-278e764e-49e8-49ce-bd77-aaebaece18aa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd27f26d6061f41848f52cd6d4416a97f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:16.211384 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-278e764e-49e8-49ce-bd77-aaebaece18aa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd27f26d6061f41848f52cd6d4416a97f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:16.212579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-278e764e-49e8-49ce-bd77-aaebaece18aa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd27f26d6061f41848f52cd6d4416a97f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:16.222602 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-278e764e-49e8-49ce-bd77-aaebaece18aa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd27f26d6061f41848f52cd6d4416a97f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:16.239130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 831/1655] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:48:15 2022] DELETE /identity/v3/users/d27f26d6061f41848f52cd6d4416a97f => generated 0 bytes in 675 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:16.411089 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:17.276869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4b2c77-d8f3-40e0-8dab-4ab5a5e40efb admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:17.277956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3c4b2c77-d8f3-40e0-8dab-4ab5a5e40efb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:17.308366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4b2c77-d8f3-40e0-8dab-4ab5a5e40efb admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:17.461797 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58585334-fe2d-48de-8f32-cedf36c23d1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:17.465032 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58585334-fe2d-48de-8f32-cedf36c23d1d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:17.468050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58585334-fe2d-48de-8f32-cedf36c23d1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:17.469876 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58585334-fe2d-48de-8f32-cedf36c23d1d admin admin] PATH_INFO: `/v3/users/e368482ff3834c0fa4a5579e29ae586e` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:17.472525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58585334-fe2d-48de-8f32-cedf36c23d1d admin admin] RBAC: Authorizing `identity:delete_user(user_id=e368482ff3834c0fa4a5579e29ae586e)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:17.487723 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: WARNING py.warnings [None req-58585334-fe2d-48de-8f32-cedf36c23d1d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 03:48:17.487723 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: warnings.warn(msg) Sep 16 03:48:17.487723 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: Sep 16 03:48:17.494003 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58585334-fe2d-48de-8f32-cedf36c23d1d admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:17.600739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278e764e-49e8-49ce-bd77-aaebaece18aa admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:17.602878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-278e764e-49e8-49ce-bd77-aaebaece18aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:17.625896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278e764e-49e8-49ce-bd77-aaebaece18aa admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:17.651295 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ab0656-63d1-4c44-bdf9-688b0af1cc04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:17.652098 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ab0656-63d1-4c44-bdf9-688b0af1cc04 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:17.653483 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ab0656-63d1-4c44-bdf9-688b0af1cc04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:17.653866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ab0656-63d1-4c44-bdf9-688b0af1cc04 admin admin] PATH_INFO: `/v3/projects/4e9e0e53ce3f4f2bbcc0227e9d118db0` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:17.673940 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ab0656-63d1-4c44-bdf9-688b0af1cc04 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4e9e0e53ce3f4f2bbcc0227e9d118db0)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:17.675236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: WARNING py.warnings [None req-26ab0656-63d1-4c44-bdf9-688b0af1cc04 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 03:48:17.675236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: warnings.warn(msg) Sep 16 03:48:17.675236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: Sep 16 03:48:17.678284 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ab0656-63d1-4c44-bdf9-688b0af1cc04 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:17.829815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-58585334-fe2d-48de-8f32-cedf36c23d1d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e368482ff3834c0fa4a5579e29ae586e'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:17.954145 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-58585334-fe2d-48de-8f32-cedf36c23d1d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e368482ff3834c0fa4a5579e29ae586e'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:18.024516 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-26ab0656-63d1-4c44-bdf9-688b0af1cc04 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4e9e0e53ce3f4f2bbcc0227e9d118db0', 'request_id': 'req-26ab0656-63d1-4c44-bdf9-688b0af1cc04'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:18.029480 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-58585334-fe2d-48de-8f32-cedf36c23d1d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e368482ff3834c0fa4a5579e29ae586e'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:18.034006 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-58585334-fe2d-48de-8f32-cedf36c23d1d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e368482ff3834c0fa4a5579e29ae586e'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:18.099738 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-58585334-fe2d-48de-8f32-cedf36c23d1d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e368482ff3834c0fa4a5579e29ae586e'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:18.103078 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-26ab0656-63d1-4c44-bdf9-688b0af1cc04 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4e9e0e53ce3f4f2bbcc0227e9d118db0', 'request_id': 'req-26ab0656-63d1-4c44-bdf9-688b0af1cc04'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:18.127758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:18.128419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 825/1656] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:48:17 2022] DELETE /identity/v3/users/e368482ff3834c0fa4a5579e29ae586e => generated 0 bytes in 852 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:18.138380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6379df07-21f0-4107-a5c9-34393519c054 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:18.140402 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6379df07-21f0-4107-a5c9-34393519c054 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:18.140839 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6379df07-21f0-4107-a5c9-34393519c054 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:18.182788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:18.187905 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-6379df07-21f0-4107-a5c9-34393519c054 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:18.192532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-6379df07-21f0-4107-a5c9-34393519c054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:18.220254 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-26ab0656-63d1-4c44-bdf9-688b0af1cc04 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4e9e0e53ce3f4f2bbcc0227e9d118db0', 'request_id': 'req-26ab0656-63d1-4c44-bdf9-688b0af1cc04'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:18.249845 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-26ab0656-63d1-4c44-bdf9-688b0af1cc04 admin admin] The token cache is being invalidate because project 4e9e0e53ce3f4f2bbcc0227e9d118db0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:48:18.250286 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-26ab0656-63d1-4c44-bdf9-688b0af1cc04 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:18.251460 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-26ab0656-63d1-4c44-bdf9-688b0af1cc04 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:18.256437 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 832/1657] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:48:17 2022] DELETE /identity/v3/projects/4e9e0e53ce3f4f2bbcc0227e9d118db0 => generated 0 bytes in 658 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:48:18.264443 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f985e1-3b94-4d5c-af23-f8050b9100b3 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:18.264443 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f985e1-3b94-4d5c-af23-f8050b9100b3 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:18.264443 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f985e1-3b94-4d5c-af23-f8050b9100b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:18.355227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-41f985e1-3b94-4d5c-af23-f8050b9100b3 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:18.358455 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-41f985e1-3b94-4d5c-af23-f8050b9100b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:18.423226 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 826/1658] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 284 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:18.478134 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6379df07-21f0-4107-a5c9-34393519c054 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:18.480384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-6379df07-21f0-4107-a5c9-34393519c054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:18.621280 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 833/1659] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 361 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:18.631560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a53a2b4-9b85-455c-95ca-222d6c1667b1 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:18.631971 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a53a2b4-9b85-455c-95ca-222d6c1667b1 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:18.632241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a53a2b4-9b85-455c-95ca-222d6c1667b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:18.683424 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-0a53a2b4-9b85-455c-95ca-222d6c1667b1 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:18.684660 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0a53a2b4-9b85-455c-95ca-222d6c1667b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:18.689230 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6379df07-21f0-4107-a5c9-34393519c054 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:18.705536 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c84e43-322d-4c5c-a367-55d112ab7f94 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:18.706553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c84e43-322d-4c5c-a367-55d112ab7f94 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:18.716664 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c84e43-322d-4c5c-a367-55d112ab7f94 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:18.716664 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c84e43-322d-4c5c-a367-55d112ab7f94 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:18.720743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 834/1660] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:18.721745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a53a2b4-9b85-455c-95ca-222d6c1667b1 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:18.744266 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a53a2b4-9b85-455c-95ca-222d6c1667b1 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:18.757398 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c84e43-322d-4c5c-a367-55d112ab7f94 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:18.759344 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c84e43-322d-4c5c-a367-55d112ab7f94 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:18.759455 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58264b3a-8a2b-4d81-9a63-33229b4202ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:18.760160 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58264b3a-8a2b-4d81-9a63-33229b4202ca admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:18.760349 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58264b3a-8a2b-4d81-9a63-33229b4202ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:18.760628 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58264b3a-8a2b-4d81-9a63-33229b4202ca admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:18.761290 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58264b3a-8a2b-4d81-9a63-33229b4202ca admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:48:18.761599 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58264b3a-8a2b-4d81-9a63-33229b4202ca admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:18.763916 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58264b3a-8a2b-4d81-9a63-33229b4202ca admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:18.804385 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 835/1661] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:48:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:48:18.805189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58264b3a-8a2b-4d81-9a63-33229b4202ca admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:18.814113 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 827/1662] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:18.830060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58264b3a-8a2b-4d81-9a63-33229b4202ca admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:18.841742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74401cd8-2b68-4561-8fce-71004e3adc30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:18.842996 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74401cd8-2b68-4561-8fce-71004e3adc30 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:18.843286 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74401cd8-2b68-4561-8fce-71004e3adc30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:18.843563 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74401cd8-2b68-4561-8fce-71004e3adc30 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:18.844211 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74401cd8-2b68-4561-8fce-71004e3adc30 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:18.846169 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74401cd8-2b68-4561-8fce-71004e3adc30 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:18.993328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 836/1663] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:18 2022] POST /identity/v3/projects => generated 369 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:18.997679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c84e43-322d-4c5c-a367-55d112ab7f94 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:19.017618 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c84e43-322d-4c5c-a367-55d112ab7f94 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:19.159772 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942135ad-da78-4b4e-8761-ac2cbd6819d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:19.164108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942135ad-da78-4b4e-8761-ac2cbd6819d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:19.164108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942135ad-da78-4b4e-8761-ac2cbd6819d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:19.164108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942135ad-da78-4b4e-8761-ac2cbd6819d6 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:19.164108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942135ad-da78-4b4e-8761-ac2cbd6819d6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:19.164108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942135ad-da78-4b4e-8761-ac2cbd6819d6 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:19.286771 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 828/1664] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:18 2022] POST /identity/v3/users => generated 285 bytes in 289 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:19.292488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74401cd8-2b68-4561-8fce-71004e3adc30 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:19.317888 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74401cd8-2b68-4561-8fce-71004e3adc30 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:19.329182 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e865f461-f22d-4dbd-adf5-682bf01e9bf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:19.330186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e865f461-f22d-4dbd-adf5-682bf01e9bf5 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:19.330658 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e865f461-f22d-4dbd-adf5-682bf01e9bf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:19.331161 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e865f461-f22d-4dbd-adf5-682bf01e9bf5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:19.331994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e865f461-f22d-4dbd-adf5-682bf01e9bf5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:19.335117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e865f461-f22d-4dbd-adf5-682bf01e9bf5 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:19.373988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 837/1665] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:19 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:19.379971 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942135ad-da78-4b4e-8761-ac2cbd6819d6 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:19.394195 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942135ad-da78-4b4e-8761-ac2cbd6819d6 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:19.395882 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e865f461-f22d-4dbd-adf5-682bf01e9bf5 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:19.399203 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e865f461-f22d-4dbd-adf5-682bf01e9bf5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:19.418183 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9289c0-d987-4861-ae55-cd07e13b66a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:19.418812 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9289c0-d987-4861-ae55-cd07e13b66a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:19.419122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9289c0-d987-4861-ae55-cd07e13b66a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:19.419403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9289c0-d987-4861-ae55-cd07e13b66a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:19.420061 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9289c0-d987-4861-ae55-cd07e13b66a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:19.422926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9289c0-d987-4861-ae55-cd07e13b66a7 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:19.433513 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e865f461-f22d-4dbd-adf5-682bf01e9bf5 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:19.444466 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b55d641-f603-4935-b23a-b0f7d58cdb00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:19.448402 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b55d641-f603-4935-b23a-b0f7d58cdb00 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:19.448402 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b55d641-f603-4935-b23a-b0f7d58cdb00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:19.448402 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b55d641-f603-4935-b23a-b0f7d58cdb00 admin admin] PATH_INFO: `/v3/users/734b27ebb07948d4839b08ba71b7dc87` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:19.453778 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 829/1666] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:19 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:48:19.462938 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b55d641-f603-4935-b23a-b0f7d58cdb00 admin admin] RBAC: Authorizing `identity:delete_user(user_id=734b27ebb07948d4839b08ba71b7dc87)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:19.466099 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b55d641-f603-4935-b23a-b0f7d58cdb00 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:19.476345 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:INSERT=3,keystone:DELETE=8 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:19.485275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9289c0-d987-4861-ae55-cd07e13b66a7 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:19.526805 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9289c0-d987-4861-ae55-cd07e13b66a7 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:19.536887 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d055524-dd02-4517-97e1-d4e978ecb83d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:19.538811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d055524-dd02-4517-97e1-d4e978ecb83d admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:19.539307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d055524-dd02-4517-97e1-d4e978ecb83d admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:19.539737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d055524-dd02-4517-97e1-d4e978ecb83d admin admin] PATH_INFO: `/v3/projects/697a44c6b72645f69f92dd371730686d/users/244020b97239487188143dd58c215947/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:19.589824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d055524-dd02-4517-97e1-d4e978ecb83d admin admin] RBAC: Authorizing `identity:create_grant(project_id=697a44c6b72645f69f92dd371730686d, user_id=244020b97239487188143dd58c215947, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:19.592698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d055524-dd02-4517-97e1-d4e978ecb83d admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:19.730332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 830/1667] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:19 2022] PUT /identity/v3/projects/697a44c6b72645f69f92dd371730686d/users/244020b97239487188143dd58c215947/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:19.743596 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e93b277-a818-4dc7-ac3c-b5a227473a38 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:19.745987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e93b277-a818-4dc7-ac3c-b5a227473a38 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:19.746767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e93b277-a818-4dc7-ac3c-b5a227473a38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:19.809881 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-5e93b277-a818-4dc7-ac3c-b5a227473a38 None None] MFA Rules not processed for user `244020b97239487188143dd58c215947`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:19.811637 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-5e93b277-a818-4dc7-ac3c-b5a227473a38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:19.828067 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8b55d641-f603-4935-b23a-b0f7d58cdb00 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '734b27ebb07948d4839b08ba71b7dc87'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:19.858883 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8b55d641-f603-4935-b23a-b0f7d58cdb00 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '734b27ebb07948d4839b08ba71b7dc87'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:19.938217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 831/1668] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:19 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:19.947757 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e93b277-a818-4dc7-ac3c-b5a227473a38 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:19.965871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8b55d641-f603-4935-b23a-b0f7d58cdb00 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '734b27ebb07948d4839b08ba71b7dc87'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:19.966715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8b55d641-f603-4935-b23a-b0f7d58cdb00 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '734b27ebb07948d4839b08ba71b7dc87'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:19.978161 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8b55d641-f603-4935-b23a-b0f7d58cdb00 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '734b27ebb07948d4839b08ba71b7dc87'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:19.993449 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 838/1669] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:48:19 2022] DELETE /identity/v3/users/734b27ebb07948d4839b08ba71b7dc87 => generated 0 bytes in 598 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:20.001813 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d768c135-44b9-4907-a39f-9bdb4483645a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:20.002512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d768c135-44b9-4907-a39f-9bdb4483645a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:20.003294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d768c135-44b9-4907-a39f-9bdb4483645a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:20.068898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-d768c135-44b9-4907-a39f-9bdb4483645a None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:20.071961 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d768c135-44b9-4907-a39f-9bdb4483645a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:20.132691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e93b277-a818-4dc7-ac3c-b5a227473a38 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:20.184362 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:20.208358 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 839/1670] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:20.221209 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d768c135-44b9-4907-a39f-9bdb4483645a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:20.222777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d768c135-44b9-4907-a39f-9bdb4483645a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:20.261670 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd368ed-39e2-4f68-ba17-880da11a5c57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:20.262577 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd368ed-39e2-4f68-ba17-880da11a5c57 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:20.262982 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd368ed-39e2-4f68-ba17-880da11a5c57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:20.263358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd368ed-39e2-4f68-ba17-880da11a5c57 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:20.264674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd368ed-39e2-4f68-ba17-880da11a5c57 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:20.266761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd368ed-39e2-4f68-ba17-880da11a5c57 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:20.351225 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d768c135-44b9-4907-a39f-9bdb4483645a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:20.356170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e0b48f-c126-4aae-a6f8-81ef1fcc4173 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:20.357091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e0b48f-c126-4aae-a6f8-81ef1fcc4173 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:20.358207 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e0b48f-c126-4aae-a6f8-81ef1fcc4173 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:20.358843 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e0b48f-c126-4aae-a6f8-81ef1fcc4173 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:20.374384 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e0b48f-c126-4aae-a6f8-81ef1fcc4173 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:20.376937 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e0b48f-c126-4aae-a6f8-81ef1fcc4173 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:20.395326 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 840/1671] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:48:20.454002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 832/1672] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:19 2022] POST /identity/v3/projects => generated 367 bytes in 510 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:20.458237 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e0b48f-c126-4aae-a6f8-81ef1fcc4173 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:20.459986 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b4e0b48f-c126-4aae-a6f8-81ef1fcc4173 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:20.482281 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e0b48f-c126-4aae-a6f8-81ef1fcc4173 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:20.579308 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21dc232b-c4e9-4941-b299-1550edf1ab2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:20.579872 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21dc232b-c4e9-4941-b299-1550edf1ab2b admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:20.581351 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21dc232b-c4e9-4941-b299-1550edf1ab2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:20.581643 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21dc232b-c4e9-4941-b299-1550edf1ab2b admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:20.582453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21dc232b-c4e9-4941-b299-1550edf1ab2b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:20.584457 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21dc232b-c4e9-4941-b299-1550edf1ab2b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:20.711722 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 841/1673] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:20 2022] POST /identity/v3/users => generated 290 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:20.715396 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd368ed-39e2-4f68-ba17-880da11a5c57 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:20.735185 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd368ed-39e2-4f68-ba17-880da11a5c57 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:20.751173 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664d8809-56a8-43ae-97bf-1353cf7507fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:20.754475 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664d8809-56a8-43ae-97bf-1353cf7507fd admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:20.754475 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664d8809-56a8-43ae-97bf-1353cf7507fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:20.754475 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664d8809-56a8-43ae-97bf-1353cf7507fd admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:20.754475 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664d8809-56a8-43ae-97bf-1353cf7507fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:20.755047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664d8809-56a8-43ae-97bf-1353cf7507fd admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:20.786063 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 833/1674] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:20 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:20.792638 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21dc232b-c4e9-4941-b299-1550edf1ab2b admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:20.804157 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21dc232b-c4e9-4941-b299-1550edf1ab2b admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:20.812712 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d789b7b-7810-4404-bc42-7a53b8f81b45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:20.813711 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d789b7b-7810-4404-bc42-7a53b8f81b45 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:20.814295 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d789b7b-7810-4404-bc42-7a53b8f81b45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:20.814722 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d789b7b-7810-4404-bc42-7a53b8f81b45 admin admin] PATH_INFO: `/v3/projects/479b376d470947e2a284f08e804ca174/users/9ce644bc53ba49aa80df3ec385a2768b/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:20.842697 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d789b7b-7810-4404-bc42-7a53b8f81b45 admin admin] RBAC: Authorizing `identity:create_grant(project_id=479b376d470947e2a284f08e804ca174, user_id=9ce644bc53ba49aa80df3ec385a2768b, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:20.848652 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d789b7b-7810-4404-bc42-7a53b8f81b45 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:20.971043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664d8809-56a8-43ae-97bf-1353cf7507fd admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:20.976165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 842/1675] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:20 2022] PUT /identity/v3/projects/479b376d470947e2a284f08e804ca174/users/9ce644bc53ba49aa80df3ec385a2768b/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:20.985192 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664d8809-56a8-43ae-97bf-1353cf7507fd admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:21.113013 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccf5371-4b29-4f23-a565-ab98893d2b21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:21.113656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccf5371-4b29-4f23-a565-ab98893d2b21 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:21.117194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccf5371-4b29-4f23-a565-ab98893d2b21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:21.117194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccf5371-4b29-4f23-a565-ab98893d2b21 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:21.117194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccf5371-4b29-4f23-a565-ab98893d2b21 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:21.117194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccf5371-4b29-4f23-a565-ab98893d2b21 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:21.141051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d789b7b-7810-4404-bc42-7a53b8f81b45 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:21.146648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 834/1676] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:20 2022] GET /identity/v3/roles => generated 1420 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:21.160520 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d789b7b-7810-4404-bc42-7a53b8f81b45 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:21.172940 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850651cc-b201-453f-ab1d-4fb7591c6671 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:21.173644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850651cc-b201-453f-ab1d-4fb7591c6671 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:21.173929 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850651cc-b201-453f-ab1d-4fb7591c6671 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:21.174210 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850651cc-b201-453f-ab1d-4fb7591c6671 admin admin] PATH_INFO: `/v3/projects/479b376d470947e2a284f08e804ca174/users/9ce644bc53ba49aa80df3ec385a2768b/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:21.178086 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850651cc-b201-453f-ab1d-4fb7591c6671 admin admin] RBAC: Authorizing `identity:create_grant(project_id=479b376d470947e2a284f08e804ca174, user_id=9ce644bc53ba49aa80df3ec385a2768b, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:21.180997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850651cc-b201-453f-ab1d-4fb7591c6671 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:21.208126 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 843/1677] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:21 2022] PUT /identity/v3/projects/479b376d470947e2a284f08e804ca174/users/9ce644bc53ba49aa80df3ec385a2768b/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:21.219738 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9533925-d005-4895-9cea-a7d606fe58b1 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:21.220182 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9533925-d005-4895-9cea-a7d606fe58b1 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:21.220602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9533925-d005-4895-9cea-a7d606fe58b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:21.282620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-b9533925-d005-4895-9cea-a7d606fe58b1 None None] MFA Rules not processed for user `9ce644bc53ba49aa80df3ec385a2768b`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:21.284142 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b9533925-d005-4895-9cea-a7d606fe58b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:21.435374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850651cc-b201-453f-ab1d-4fb7591c6671 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:21.440376 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-850651cc-b201-453f-ab1d-4fb7591c6671 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:21.554236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:21.591209 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 835/1678] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:21 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 373 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:21.613044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9533925-d005-4895-9cea-a7d606fe58b1 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:21.613044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b9533925-d005-4895-9cea-a7d606fe58b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:21.621693 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850651cc-b201-453f-ab1d-4fb7591c6671 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:21.635337 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6962a25-e976-4776-bfc8-6afa9eb318cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:21.636003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6962a25-e976-4776-bfc8-6afa9eb318cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:21.636287 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6962a25-e976-4776-bfc8-6afa9eb318cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:21.636564 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6962a25-e976-4776-bfc8-6afa9eb318cd admin admin] PATH_INFO: `/v3/projects/ef5d958ed6b74751b951c2fb90804692` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:21.640455 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6962a25-e976-4776-bfc8-6afa9eb318cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=ef5d958ed6b74751b951c2fb90804692)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:21.644916 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6962a25-e976-4776-bfc8-6afa9eb318cd admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:21.792408 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9533925-d005-4895-9cea-a7d606fe58b1 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:21.797512 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23acc1e-db60-4101-a1c8-68bab4b4eabd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:21.798148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23acc1e-db60-4101-a1c8-68bab4b4eabd service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:21.798435 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23acc1e-db60-4101-a1c8-68bab4b4eabd service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:21.798720 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23acc1e-db60-4101-a1c8-68bab4b4eabd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:21.818353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23acc1e-db60-4101-a1c8-68bab4b4eabd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:21.819860 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23acc1e-db60-4101-a1c8-68bab4b4eabd service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:21.853705 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c6962a25-e976-4776-bfc8-6afa9eb318cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ef5d958ed6b74751b951c2fb90804692', 'request_id': 'req-c6962a25-e976-4776-bfc8-6afa9eb318cd'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:21.856060 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 836/1679] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:21 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 252 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:48:21.863955 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c6962a25-e976-4776-bfc8-6afa9eb318cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ef5d958ed6b74751b951c2fb90804692', 'request_id': 'req-c6962a25-e976-4776-bfc8-6afa9eb318cd'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:21.950261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c6962a25-e976-4776-bfc8-6afa9eb318cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ef5d958ed6b74751b951c2fb90804692', 'request_id': 'req-c6962a25-e976-4776-bfc8-6afa9eb318cd'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:21.982413 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23acc1e-db60-4101-a1c8-68bab4b4eabd service cinder] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:21.983989 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-f23acc1e-db60-4101-a1c8-68bab4b4eabd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:22.004313 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c6962a25-e976-4776-bfc8-6afa9eb318cd admin admin] The token cache is being invalidate because project ef5d958ed6b74751b951c2fb90804692 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:48:22.004700 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c6962a25-e976-4776-bfc8-6afa9eb318cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:22.006710 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c6962a25-e976-4776-bfc8-6afa9eb318cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:22.014230 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 844/1680] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:48:21 2022] DELETE /identity/v3/projects/ef5d958ed6b74751b951c2fb90804692 => generated 0 bytes in 579 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:48:22.016512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f329e431-b24c-482c-96f7-e01d6add93f1 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:22.021090 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f329e431-b24c-482c-96f7-e01d6add93f1 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:22.021269 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23acc1e-db60-4101-a1c8-68bab4b4eabd service cinder] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:22.021539 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f329e431-b24c-482c-96f7-e01d6add93f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:22.107808 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f329e431-b24c-482c-96f7-e01d6add93f1 None None] MFA Rules not processed for user `8f410d691e364880b13bb9144bbc5e7d`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:22.109936 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f329e431-b24c-482c-96f7-e01d6add93f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:22.137644 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f743e983-1f97-44ca-835b-304822825aed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:22.138523 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f743e983-1f97-44ca-835b-304822825aed service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:22.139845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f743e983-1f97-44ca-835b-304822825aed service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:22.140301 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f743e983-1f97-44ca-835b-304822825aed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:22.143959 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-f743e983-1f97-44ca-835b-304822825aed service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:22.188404 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:22.275113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 845/1681] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:22 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 260 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:22.276689 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9927d31d-1958-4542-8886-6bf26c3aaf25 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:22.277130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9927d31d-1958-4542-8886-6bf26c3aaf25 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:22.277584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9927d31d-1958-4542-8886-6bf26c3aaf25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:22.286087 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f743e983-1f97-44ca-835b-304822825aed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:22.287624 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f743e983-1f97-44ca-835b-304822825aed service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:22.314711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 837/1682] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:21 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:48:22.315833 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f743e983-1f97-44ca-835b-304822825aed service cinder] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:22.317496 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-f743e983-1f97-44ca-835b-304822825aed service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:22.324511 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9927d31d-1958-4542-8886-6bf26c3aaf25 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:22.324511 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9927d31d-1958-4542-8886-6bf26c3aaf25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:22.486851 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f743e983-1f97-44ca-835b-304822825aed service cinder] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:22.505973 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 846/1683] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:22.507047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9927d31d-1958-4542-8886-6bf26c3aaf25 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:22.508517 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9927d31d-1958-4542-8886-6bf26c3aaf25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:22.510083 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d310c664-c731-4f1d-9f5e-1b11915e25da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:22.510788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d310c664-c731-4f1d-9f5e-1b11915e25da admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:22.511111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d310c664-c731-4f1d-9f5e-1b11915e25da admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:22.511394 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d310c664-c731-4f1d-9f5e-1b11915e25da admin admin] PATH_INFO: `/v3/projects/e1f32c865ae54bed834184981050369a` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:22.519980 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d310c664-c731-4f1d-9f5e-1b11915e25da admin admin] RBAC: Authorizing `identity:delete_project(project_id=e1f32c865ae54bed834184981050369a)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:22.521955 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d310c664-c731-4f1d-9f5e-1b11915e25da admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:22.701783 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9927d31d-1958-4542-8886-6bf26c3aaf25 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:22.736216 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d310c664-c731-4f1d-9f5e-1b11915e25da admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e1f32c865ae54bed834184981050369a', 'request_id': 'req-d310c664-c731-4f1d-9f5e-1b11915e25da'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:22.821394 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92883ad7-e23d-4a39-8e0c-58fd2bc6698d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:22.822419 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92883ad7-e23d-4a39-8e0c-58fd2bc6698d service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:22.822710 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92883ad7-e23d-4a39-8e0c-58fd2bc6698d service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:22.823019 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92883ad7-e23d-4a39-8e0c-58fd2bc6698d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:22.879749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d310c664-c731-4f1d-9f5e-1b11915e25da admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e1f32c865ae54bed834184981050369a', 'request_id': 'req-d310c664-c731-4f1d-9f5e-1b11915e25da'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:22.900402 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d310c664-c731-4f1d-9f5e-1b11915e25da admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e1f32c865ae54bed834184981050369a', 'request_id': 'req-d310c664-c731-4f1d-9f5e-1b11915e25da'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:22.939761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d310c664-c731-4f1d-9f5e-1b11915e25da admin admin] The token cache is being invalidate because project e1f32c865ae54bed834184981050369a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:48:22.942272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d310c664-c731-4f1d-9f5e-1b11915e25da admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:22.943596 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d310c664-c731-4f1d-9f5e-1b11915e25da admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:22.950151 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 838/1684] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:48:22 2022] DELETE /identity/v3/projects/e1f32c865ae54bed834184981050369a => generated 0 bytes in 634 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:22.955260 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9995774b-ebd6-4203-9e95-6826520e2e77 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:22.956682 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9995774b-ebd6-4203-9e95-6826520e2e77 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:22.958368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9995774b-ebd6-4203-9e95-6826520e2e77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:22.982071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92883ad7-e23d-4a39-8e0c-58fd2bc6698d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:22.991952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92883ad7-e23d-4a39-8e0c-58fd2bc6698d service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:23.013777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-9995774b-ebd6-4203-9e95-6826520e2e77 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:23.030948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-9995774b-ebd6-4203-9e95-6826520e2e77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:23.031827 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 847/1685] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:22 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 525 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:23.033778 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92883ad7-e23d-4a39-8e0c-58fd2bc6698d service nova] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:23.169809 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 839/1686] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:23.169809 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9995774b-ebd6-4203-9e95-6826520e2e77 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:23.170751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92883ad7-e23d-4a39-8e0c-58fd2bc6698d service nova] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:23.176460 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa706139-4ebb-4859-b569-27525775d592 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:23.177274 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa706139-4ebb-4859-b569-27525775d592 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:23.177631 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa706139-4ebb-4859-b569-27525775d592 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:23.177980 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa706139-4ebb-4859-b569-27525775d592 admin admin] PATH_INFO: `/v3/users/c4679421808b42cdb68b1a02a2a0675b` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:23.179411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa706139-4ebb-4859-b569-27525775d592 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c4679421808b42cdb68b1a02a2a0675b)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:23.183440 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa706139-4ebb-4859-b569-27525775d592 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:23.190526 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9995774b-ebd6-4203-9e95-6826520e2e77 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:23.197159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410c81d2-0ad9-44de-a504-f2b61c0e10ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:23.197803 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410c81d2-0ad9-44de-a504-f2b61c0e10ff admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:23.204282 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410c81d2-0ad9-44de-a504-f2b61c0e10ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:23.204282 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410c81d2-0ad9-44de-a504-f2b61c0e10ff admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:23.204282 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410c81d2-0ad9-44de-a504-f2b61c0e10ff admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:48:23.204282 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410c81d2-0ad9-44de-a504-f2b61c0e10ff admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:23.205304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410c81d2-0ad9-44de-a504-f2b61c0e10ff admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:23.222425 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 840/1687] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:48:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:48:23.223654 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410c81d2-0ad9-44de-a504-f2b61c0e10ff admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:23.238358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410c81d2-0ad9-44de-a504-f2b61c0e10ff admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:23.249330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a058f76e-e98f-4303-9a39-d26921858cb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:23.250104 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a058f76e-e98f-4303-9a39-d26921858cb1 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:23.250673 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a058f76e-e98f-4303-9a39-d26921858cb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:23.251076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a058f76e-e98f-4303-9a39-d26921858cb1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:23.252785 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a058f76e-e98f-4303-9a39-d26921858cb1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:23.255391 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a058f76e-e98f-4303-9a39-d26921858cb1 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:23.361161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 841/1688] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:23 2022] POST /identity/v3/projects => generated 365 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:23.363226 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a058f76e-e98f-4303-9a39-d26921858cb1 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:23.365420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:23.494875 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=87,keystone:INSERT=6,keystone:DELETE=7 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:23.526936 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a058f76e-e98f-4303-9a39-d26921858cb1 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:23.532161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1161418c-b708-40a4-b460-8c50d8dc77d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:23.532161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1161418c-b708-40a4-b460-8c50d8dc77d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:23.537582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1161418c-b708-40a4-b460-8c50d8dc77d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:23.537582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1161418c-b708-40a4-b460-8c50d8dc77d8 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:23.537582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1161418c-b708-40a4-b460-8c50d8dc77d8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:23.537582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1161418c-b708-40a4-b460-8c50d8dc77d8 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:23.567246 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fa706139-4ebb-4859-b569-27525775d592 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c4679421808b42cdb68b1a02a2a0675b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:23.604461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fa706139-4ebb-4859-b569-27525775d592 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c4679421808b42cdb68b1a02a2a0675b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:23.675503 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 842/1689] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:23 2022] POST /identity/v3/users => generated 283 bytes in 311 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:23.677121 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1161418c-b708-40a4-b460-8c50d8dc77d8 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:23.691899 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fa706139-4ebb-4859-b569-27525775d592 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c4679421808b42cdb68b1a02a2a0675b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:23.692557 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fa706139-4ebb-4859-b569-27525775d592 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c4679421808b42cdb68b1a02a2a0675b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:23.707562 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fa706139-4ebb-4859-b569-27525775d592 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c4679421808b42cdb68b1a02a2a0675b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:23.721199 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1161418c-b708-40a4-b460-8c50d8dc77d8 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:23.732037 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 848/1690] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:48:23 2022] DELETE /identity/v3/users/c4679421808b42cdb68b1a02a2a0675b => generated 0 bytes in 699 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:23.734597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599a45c6-411a-4f33-95a7-854e0f02988d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:23.735788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599a45c6-411a-4f33-95a7-854e0f02988d admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:23.736318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599a45c6-411a-4f33-95a7-854e0f02988d admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:23.736795 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599a45c6-411a-4f33-95a7-854e0f02988d admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:23.738003 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599a45c6-411a-4f33-95a7-854e0f02988d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:23.740145 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599a45c6-411a-4f33-95a7-854e0f02988d admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:23.757248 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 843/1691] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:23 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:48:23.761286 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa706139-4ebb-4859-b569-27525775d592 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:23.763522 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fa706139-4ebb-4859-b569-27525775d592 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:23.906407 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa706139-4ebb-4859-b569-27525775d592 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:23.948601 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f603750b-f550-4e0c-b4cf-9c6846b0a6ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:23.949308 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f603750b-f550-4e0c-b4cf-9c6846b0a6ac admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:23.950219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f603750b-f550-4e0c-b4cf-9c6846b0a6ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:23.950516 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f603750b-f550-4e0c-b4cf-9c6846b0a6ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:23.954006 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f603750b-f550-4e0c-b4cf-9c6846b0a6ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:23.959698 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f603750b-f550-4e0c-b4cf-9c6846b0a6ac admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:23.983240 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 849/1692] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:23 2022] GET /identity/v3/roles => generated 1420 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:23.987754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599a45c6-411a-4f33-95a7-854e0f02988d admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:24.019899 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599a45c6-411a-4f33-95a7-854e0f02988d admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:24.030841 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4c8717-1891-4553-86a6-2909c44b856e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:24.031981 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4c8717-1891-4553-86a6-2909c44b856e admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:24.032435 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4c8717-1891-4553-86a6-2909c44b856e admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:24.032880 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4c8717-1891-4553-86a6-2909c44b856e admin admin] PATH_INFO: `/v3/projects/9a94dc3204094677b9ee95ab41055bf0/users/714787130ac848c69fe72d7da4cd0263/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:24.070464 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4c8717-1891-4553-86a6-2909c44b856e admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a94dc3204094677b9ee95ab41055bf0, user_id=714787130ac848c69fe72d7da4cd0263, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:24.082126 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4c8717-1891-4553-86a6-2909c44b856e admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:24.170026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 844/1693] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:23 2022] PUT /identity/v3/projects/9a94dc3204094677b9ee95ab41055bf0/users/714787130ac848c69fe72d7da4cd0263/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:24.182449 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302f9faa-9d17-430e-bb14-234271d2d84f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:24.183026 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302f9faa-9d17-430e-bb14-234271d2d84f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:24.183640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302f9faa-9d17-430e-bb14-234271d2d84f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:24.262512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-302f9faa-9d17-430e-bb14-234271d2d84f None None] MFA Rules not processed for user `714787130ac848c69fe72d7da4cd0263`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:24.265782 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-302f9faa-9d17-430e-bb14-234271d2d84f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:24.482604 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 850/1694] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:24 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 298 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:24.484806 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4c8717-1891-4553-86a6-2909c44b856e admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:24.705795 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:24.797942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4c8717-1891-4553-86a6-2909c44b856e admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:24.817310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c7662c-0634-4099-9974-37b3938f404c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:24.818055 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c7662c-0634-4099-9974-37b3938f404c admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:24.818447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c7662c-0634-4099-9974-37b3938f404c admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:24.818755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c7662c-0634-4099-9974-37b3938f404c admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:24.819595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c7662c-0634-4099-9974-37b3938f404c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:24.825322 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c7662c-0634-4099-9974-37b3938f404c admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:24.986342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 845/1695] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:24 2022] POST /identity/v3/projects => generated 363 bytes in 502 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:48:24.992950 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302f9faa-9d17-430e-bb14-234271d2d84f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:25.028506 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302f9faa-9d17-430e-bb14-234271d2d84f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:25.135801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2695cb10-ddb8-4eb5-8a4e-6cc04f3e57f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:25.136525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2695cb10-ddb8-4eb5-8a4e-6cc04f3e57f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:25.136628 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2695cb10-ddb8-4eb5-8a4e-6cc04f3e57f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:25.136911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2695cb10-ddb8-4eb5-8a4e-6cc04f3e57f1 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:25.139937 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2695cb10-ddb8-4eb5-8a4e-6cc04f3e57f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:25.142020 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2695cb10-ddb8-4eb5-8a4e-6cc04f3e57f1 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:25.273935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 851/1696] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:24 2022] POST /identity/v3/users => generated 288 bytes in 282 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:25.275707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c7662c-0634-4099-9974-37b3938f404c admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:25.291580 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c7662c-0634-4099-9974-37b3938f404c admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:25.312089 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb03ab7-5bf1-4440-8978-80250ffd2e26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:25.312089 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb03ab7-5bf1-4440-8978-80250ffd2e26 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:25.312089 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb03ab7-5bf1-4440-8978-80250ffd2e26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:25.312089 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb03ab7-5bf1-4440-8978-80250ffd2e26 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:25.312432 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb03ab7-5bf1-4440-8978-80250ffd2e26 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:25.326216 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb03ab7-5bf1-4440-8978-80250ffd2e26 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:25.368093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 846/1697] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:25 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:25.370578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2695cb10-ddb8-4eb5-8a4e-6cc04f3e57f1 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:25.415052 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2695cb10-ddb8-4eb5-8a4e-6cc04f3e57f1 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:25.432765 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86246f30-e4b8-413e-9072-ed87bedfa54a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:25.433528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86246f30-e4b8-413e-9072-ed87bedfa54a admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:25.433814 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86246f30-e4b8-413e-9072-ed87bedfa54a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:25.434092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86246f30-e4b8-413e-9072-ed87bedfa54a admin admin] PATH_INFO: `/v3/projects/ef431b2d03774380a4ead2dd4ff575c7/users/867ea06d48104549b78442411aea7aab/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:25.480064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86246f30-e4b8-413e-9072-ed87bedfa54a admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef431b2d03774380a4ead2dd4ff575c7, user_id=867ea06d48104549b78442411aea7aab, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:25.491013 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86246f30-e4b8-413e-9072-ed87bedfa54a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:25.525943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:25.599136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 852/1698] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:25 2022] PUT /identity/v3/projects/ef431b2d03774380a4ead2dd4ff575c7/users/867ea06d48104549b78442411aea7aab/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:25.603266 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb03ab7-5bf1-4440-8978-80250ffd2e26 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:25.625033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb03ab7-5bf1-4440-8978-80250ffd2e26 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:25.771272 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:25.851758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd5c50b-5a02-41d0-8297-e2ce921a6e04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:25.852742 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd5c50b-5a02-41d0-8297-e2ce921a6e04 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:25.853539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd5c50b-5a02-41d0-8297-e2ce921a6e04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:25.861765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd5c50b-5a02-41d0-8297-e2ce921a6e04 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:25.862633 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd5c50b-5a02-41d0-8297-e2ce921a6e04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:25.868356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd5c50b-5a02-41d0-8297-e2ce921a6e04 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:25.913038 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 847/1699] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:25 2022] GET /identity/v3/roles => generated 1420 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:25.914305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86246f30-e4b8-413e-9072-ed87bedfa54a admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:25.936574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86246f30-e4b8-413e-9072-ed87bedfa54a admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:25.946013 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e296c014-5a9b-4945-a231-06ddf417a18e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:25.946788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e296c014-5a9b-4945-a231-06ddf417a18e admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:25.950405 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e296c014-5a9b-4945-a231-06ddf417a18e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:25.951158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e296c014-5a9b-4945-a231-06ddf417a18e admin admin] PATH_INFO: `/v3/projects/ef431b2d03774380a4ead2dd4ff575c7/users/867ea06d48104549b78442411aea7aab/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:25.966702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e296c014-5a9b-4945-a231-06ddf417a18e admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef431b2d03774380a4ead2dd4ff575c7, user_id=867ea06d48104549b78442411aea7aab, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:25.970241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e296c014-5a9b-4945-a231-06ddf417a18e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:25.994368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 853/1700] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:25 2022] PUT /identity/v3/projects/ef431b2d03774380a4ead2dd4ff575c7/users/867ea06d48104549b78442411aea7aab/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:26.005335 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c5387f-869f-4c79-bfb5-383a85736995 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:26.005763 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c5387f-869f-4c79-bfb5-383a85736995 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:26.006104 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c5387f-869f-4c79-bfb5-383a85736995 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:26.066874 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-e8c5387f-869f-4c79-bfb5-383a85736995 None None] MFA Rules not processed for user `867ea06d48104549b78442411aea7aab`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:26.069481 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e8c5387f-869f-4c79-bfb5-383a85736995 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:26.214443 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:26.308446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 848/1701] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:26 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 304 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:26.317904 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e296c014-5a9b-4945-a231-06ddf417a18e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:26.325213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e296c014-5a9b-4945-a231-06ddf417a18e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:26.437015 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c5387f-869f-4c79-bfb5-383a85736995 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:26.439499 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e8c5387f-869f-4c79-bfb5-383a85736995 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:26.460429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e296c014-5a9b-4945-a231-06ddf417a18e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:26.474787 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbbd238-7821-4e22-b602-187ad5c87f22 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:26.478655 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbbd238-7821-4e22-b602-187ad5c87f22 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:26.479004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbbd238-7821-4e22-b602-187ad5c87f22 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:26.479791 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbbd238-7821-4e22-b602-187ad5c87f22 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:26.523119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbbd238-7821-4e22-b602-187ad5c87f22 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:26.525871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbbd238-7821-4e22-b602-187ad5c87f22 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:26.580985 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 854/1702] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:26 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 264 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:26.699427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c5387f-869f-4c79-bfb5-383a85736995 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:26.713926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e55e968-a55e-4b28-806d-51917f0fda20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:26.714984 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e55e968-a55e-4b28-806d-51917f0fda20 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:26.715614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e55e968-a55e-4b28-806d-51917f0fda20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:26.716697 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e55e968-a55e-4b28-806d-51917f0fda20 admin admin] PATH_INFO: `/v3/projects/64845a46eab34d8bbbc9a4606c1a7b35` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:26.722193 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e55e968-a55e-4b28-806d-51917f0fda20 admin admin] RBAC: Authorizing `identity:delete_project(project_id=64845a46eab34d8bbbc9a4606c1a7b35)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:26.728750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e55e968-a55e-4b28-806d-51917f0fda20 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:26.958078 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-8e55e968-a55e-4b28-806d-51917f0fda20 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '64845a46eab34d8bbbc9a4606c1a7b35', 'request_id': 'req-8e55e968-a55e-4b28-806d-51917f0fda20'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:27.057284 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-8e55e968-a55e-4b28-806d-51917f0fda20 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '64845a46eab34d8bbbc9a4606c1a7b35', 'request_id': 'req-8e55e968-a55e-4b28-806d-51917f0fda20'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:27.064660 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-8e55e968-a55e-4b28-806d-51917f0fda20 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '64845a46eab34d8bbbc9a4606c1a7b35', 'request_id': 'req-8e55e968-a55e-4b28-806d-51917f0fda20'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:27.106169 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-8e55e968-a55e-4b28-806d-51917f0fda20 admin admin] The token cache is being invalidate because project 64845a46eab34d8bbbc9a4606c1a7b35 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:48:27.106169 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-8e55e968-a55e-4b28-806d-51917f0fda20 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:27.106169 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-8e55e968-a55e-4b28-806d-51917f0fda20 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:27.107379 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 849/1703] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:48:26 2022] DELETE /identity/v3/projects/64845a46eab34d8bbbc9a4606c1a7b35 => generated 0 bytes in 671 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:27.112067 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbbd238-7821-4e22-b602-187ad5c87f22 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:27.136916 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbbd238-7821-4e22-b602-187ad5c87f22 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:27.299326 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e72655c-be00-44be-84cc-8b687c022219 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:27.300142 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e72655c-be00-44be-84cc-8b687c022219 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:27.300530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e72655c-be00-44be-84cc-8b687c022219 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:27.301308 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e72655c-be00-44be-84cc-8b687c022219 admin admin] PATH_INFO: `/v3/users/c2f718ebaee24d98a51df42a33009eff` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:27.302898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e72655c-be00-44be-84cc-8b687c022219 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c2f718ebaee24d98a51df42a33009eff)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:27.305092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e72655c-be00-44be-84cc-8b687c022219 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:27.665009 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2e72655c-be00-44be-84cc-8b687c022219 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c2f718ebaee24d98a51df42a33009eff'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:27.699483 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2e72655c-be00-44be-84cc-8b687c022219 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c2f718ebaee24d98a51df42a33009eff'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:27.829634 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2e72655c-be00-44be-84cc-8b687c022219 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c2f718ebaee24d98a51df42a33009eff'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:27.831288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2e72655c-be00-44be-84cc-8b687c022219 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c2f718ebaee24d98a51df42a33009eff'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:27.847717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2e72655c-be00-44be-84cc-8b687c022219 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c2f718ebaee24d98a51df42a33009eff'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:27.872701 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 855/1704] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:48:27 2022] DELETE /identity/v3/users/c2f718ebaee24d98a51df42a33009eff => generated 0 bytes in 757 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:27.887533 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:28.254818 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894e0c9b-ed6e-41ea-a79e-0bc65f647958 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:28.254818 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894e0c9b-ed6e-41ea-a79e-0bc65f647958 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:28.254818 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894e0c9b-ed6e-41ea-a79e-0bc65f647958 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:28.334929 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-894e0c9b-ed6e-41ea-a79e-0bc65f647958 None None] MFA Rules not processed for user `c1e4846769d44451b239dee5c892b5e8`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:28.336338 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-894e0c9b-ed6e-41ea-a79e-0bc65f647958 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:28.533363 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 850/1705] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:28 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:48:28.539037 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b65db6-c85c-4d1c-b107-9c51cd4a1f99 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:28.539499 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b65db6-c85c-4d1c-b107-9c51cd4a1f99 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:28.539799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b65db6-c85c-4d1c-b107-9c51cd4a1f99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:28.565299 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb40f000-740b-44e5-bc60-1b8db92e5c23 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:28.565781 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb40f000-740b-44e5-bc60-1b8db92e5c23 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:28.566346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb40f000-740b-44e5-bc60-1b8db92e5c23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:28.631078 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-c5b65db6-c85c-4d1c-b107-9c51cd4a1f99 None None] MFA Rules not processed for user `c1e4846769d44451b239dee5c892b5e8`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:28.634312 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:28.641704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c5b65db6-c85c-4d1c-b107-9c51cd4a1f99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:28.645875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-fb40f000-740b-44e5-bc60-1b8db92e5c23 None None] MFA Rules not processed for user `ff7df9be8475405eba924b59a7bd6bb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:28.647171 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-fb40f000-740b-44e5-bc60-1b8db92e5c23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:28.656589 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 856/1706] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:28 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:28.657520 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b65db6-c85c-4d1c-b107-9c51cd4a1f99 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:28.664038 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c5b65db6-c85c-4d1c-b107-9c51cd4a1f99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:28.837304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 851/1707] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:28 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:28.838405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb40f000-740b-44e5-bc60-1b8db92e5c23 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:28.839927 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-fb40f000-740b-44e5-bc60-1b8db92e5c23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:28.841937 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b65db6-c85c-4d1c-b107-9c51cd4a1f99 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:28.858877 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7b2fed-a51d-4173-aae0-7fb492e7fea9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:28.860577 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7b2fed-a51d-4173-aae0-7fb492e7fea9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:28.860945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7b2fed-a51d-4173-aae0-7fb492e7fea9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:28.861369 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7b2fed-a51d-4173-aae0-7fb492e7fea9 admin admin] PATH_INFO: `/v3/users/714787130ac848c69fe72d7da4cd0263` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:28.863794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7b2fed-a51d-4173-aae0-7fb492e7fea9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=714787130ac848c69fe72d7da4cd0263)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:28.865733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7b2fed-a51d-4173-aae0-7fb492e7fea9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:28.965914 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:29.101946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb40f000-740b-44e5-bc60-1b8db92e5c23 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:29.142170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063a281c-234a-4724-a49f-4d25782dd280 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:29.143393 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063a281c-234a-4724-a49f-4d25782dd280 service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:29.143393 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063a281c-234a-4724-a49f-4d25782dd280 service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:29.143393 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063a281c-234a-4724-a49f-4d25782dd280 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:29.165580 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063a281c-234a-4724-a49f-4d25782dd280 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:29.167320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063a281c-234a-4724-a49f-4d25782dd280 service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:29.171267 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-ca7b2fed-a51d-4173-aae0-7fb492e7fea9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '714787130ac848c69fe72d7da4cd0263'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:29.200057 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 852/1708] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:28 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:29.201023 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063a281c-234a-4724-a49f-4d25782dd280 service nova] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:29.219639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-063a281c-234a-4724-a49f-4d25782dd280 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:29.245357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-ca7b2fed-a51d-4173-aae0-7fb492e7fea9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '714787130ac848c69fe72d7da4cd0263'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:29.262161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063a281c-234a-4724-a49f-4d25782dd280 service nova] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:29.288328 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642ee8b9-acf4-48ad-8ea5-35337b689cc4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:29.295436 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642ee8b9-acf4-48ad-8ea5-35337b689cc4 service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:29.295917 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642ee8b9-acf4-48ad-8ea5-35337b689cc4 service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:29.298474 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642ee8b9-acf4-48ad-8ea5-35337b689cc4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:29.358527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642ee8b9-acf4-48ad-8ea5-35337b689cc4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:29.366221 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642ee8b9-acf4-48ad-8ea5-35337b689cc4 service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:29.399359 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-ca7b2fed-a51d-4173-aae0-7fb492e7fea9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '714787130ac848c69fe72d7da4cd0263'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:29.399968 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-ca7b2fed-a51d-4173-aae0-7fb492e7fea9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '714787130ac848c69fe72d7da4cd0263'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:29.421345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 853/1709] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:29 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:29.425719 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-ca7b2fed-a51d-4173-aae0-7fb492e7fea9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '714787130ac848c69fe72d7da4cd0263'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:29.471394 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 857/1710] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:48:28 2022] DELETE /identity/v3/users/714787130ac848c69fe72d7da4cd0263 => generated 0 bytes in 814 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:29.475933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3a4dc2-8804-4aa2-90d3-7827c67256e9 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:29.476479 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3a4dc2-8804-4aa2-90d3-7827c67256e9 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:29.476934 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3a4dc2-8804-4aa2-90d3-7827c67256e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:29.541262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-bb3a4dc2-8804-4aa2-90d3-7827c67256e9 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:29.544090 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-bb3a4dc2-8804-4aa2-90d3-7827c67256e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:29.783000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 854/1711] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:29.790407 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0140f547-d7ab-4b52-84d8-c511e10820f7 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:29.790892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0140f547-d7ab-4b52-84d8-c511e10820f7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:29.792034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0140f547-d7ab-4b52-84d8-c511e10820f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:29.797335 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3a4dc2-8804-4aa2-90d3-7827c67256e9 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:29.800326 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-bb3a4dc2-8804-4aa2-90d3-7827c67256e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:29.864966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-0140f547-d7ab-4b52-84d8-c511e10820f7 None None] MFA Rules not processed for user `416479ba056f4612b6214cf115251269`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:29.866226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0140f547-d7ab-4b52-84d8-c511e10820f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:29.950407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3a4dc2-8804-4aa2-90d3-7827c67256e9 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:29.957490 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb1fdb3-129e-4c78-be2e-75bd4e851d31 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:29.958134 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb1fdb3-129e-4c78-be2e-75bd4e851d31 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:29.958414 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb1fdb3-129e-4c78-be2e-75bd4e851d31 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:29.958693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb1fdb3-129e-4c78-be2e-75bd4e851d31 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:29.992792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb1fdb3-129e-4c78-be2e-75bd4e851d31 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:29.994788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb1fdb3-129e-4c78-be2e-75bd4e851d31 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:30.039016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 855/1712] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 241 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:30.075073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 858/1713] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:29 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 286 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:48:30.091792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb1fdb3-129e-4c78-be2e-75bd4e851d31 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:30.093691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-7bb1fdb3-129e-4c78-be2e-75bd4e851d31 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:30.114144 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:30.274165 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb1fdb3-129e-4c78-be2e-75bd4e851d31 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:30.288651 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e7a8f1-d86e-427d-9e21-8ecfeae8d07c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:30.289516 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e7a8f1-d86e-427d-9e21-8ecfeae8d07c service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:30.289951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e7a8f1-d86e-427d-9e21-8ecfeae8d07c service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:30.290388 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e7a8f1-d86e-427d-9e21-8ecfeae8d07c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:30.316028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e7a8f1-d86e-427d-9e21-8ecfeae8d07c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:30.319474 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e7a8f1-d86e-427d-9e21-8ecfeae8d07c service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:30.365650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 856/1714] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:30 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 275 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:30.443572 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0140f547-d7ab-4b52-84d8-c511e10820f7 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:30.445400 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0140f547-d7ab-4b52-84d8-c511e10820f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:30.466275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0140f547-d7ab-4b52-84d8-c511e10820f7 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:30.474314 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7daa88a6-aee3-480e-b68e-8c10960c4fb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:30.476508 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7daa88a6-aee3-480e-b68e-8c10960c4fb1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:30.476970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7daa88a6-aee3-480e-b68e-8c10960c4fb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:30.477594 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7daa88a6-aee3-480e-b68e-8c10960c4fb1 admin admin] PATH_INFO: `/v3/users/244020b97239487188143dd58c215947` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:30.479157 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7daa88a6-aee3-480e-b68e-8c10960c4fb1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=244020b97239487188143dd58c215947)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:30.481461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7daa88a6-aee3-480e-b68e-8c10960c4fb1 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:30.695316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e7a8f1-d86e-427d-9e21-8ecfeae8d07c service nova] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:30.697237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-11e7a8f1-d86e-427d-9e21-8ecfeae8d07c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:30.732210 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e7a8f1-d86e-427d-9e21-8ecfeae8d07c service nova] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:30.744672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e5d4b2-2d71-422c-ae47-d4f984e117af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:30.747160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e5d4b2-2d71-422c-ae47-d4f984e117af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:30.747679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e5d4b2-2d71-422c-ae47-d4f984e117af admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:30.748107 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e5d4b2-2d71-422c-ae47-d4f984e117af admin admin] PATH_INFO: `/v3/projects/655dca434c2647bcb0e935bfb1dc9b42` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:30.753711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e5d4b2-2d71-422c-ae47-d4f984e117af admin admin] RBAC: Authorizing `identity:delete_project(project_id=655dca434c2647bcb0e935bfb1dc9b42)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:30.756523 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e5d4b2-2d71-422c-ae47-d4f984e117af admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:30.768397 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7daa88a6-aee3-480e-b68e-8c10960c4fb1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '244020b97239487188143dd58c215947'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:30.799570 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7daa88a6-aee3-480e-b68e-8c10960c4fb1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '244020b97239487188143dd58c215947'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:30.880175 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7daa88a6-aee3-480e-b68e-8c10960c4fb1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '244020b97239487188143dd58c215947'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:30.880979 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7daa88a6-aee3-480e-b68e-8c10960c4fb1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '244020b97239487188143dd58c215947'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:30.895815 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7daa88a6-aee3-480e-b68e-8c10960c4fb1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '244020b97239487188143dd58c215947'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:30.916357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 859/1715] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:48:30 2022] DELETE /identity/v3/users/244020b97239487188143dd58c215947 => generated 0 bytes in 473 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:30.933466 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663d0531-740c-407e-b6f8-ac060ab11119 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:30.933790 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663d0531-740c-407e-b6f8-ac060ab11119 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:30.934261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663d0531-740c-407e-b6f8-ac060ab11119 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:30.960252 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c0e5d4b2-2d71-422c-ae47-d4f984e117af admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '655dca434c2647bcb0e935bfb1dc9b42', 'request_id': 'req-c0e5d4b2-2d71-422c-ae47-d4f984e117af'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:30.980106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-663d0531-740c-407e-b6f8-ac060ab11119 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:30.986243 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-663d0531-740c-407e-b6f8-ac060ab11119 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:31.083579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 860/1716] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:31.083579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169e8ff2-75f4-4599-be90-27d1d9235fb4 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:31.083579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169e8ff2-75f4-4599-be90-27d1d9235fb4 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:31.083579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169e8ff2-75f4-4599-be90-27d1d9235fb4 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:31.084882 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 861/1717] 10.222.0.49 () {54 vars in 912 bytes} [Fri Sep 16 03:48:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:48:31.094518 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49860a35-3f58-407a-a4ec-ed88f7bcd5a5 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:31.094932 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49860a35-3f58-407a-a4ec-ed88f7bcd5a5 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:31.095318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49860a35-3f58-407a-a4ec-ed88f7bcd5a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:31.136761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c0e5d4b2-2d71-422c-ae47-d4f984e117af admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '655dca434c2647bcb0e935bfb1dc9b42', 'request_id': 'req-c0e5d4b2-2d71-422c-ae47-d4f984e117af'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:31.149703 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c0e5d4b2-2d71-422c-ae47-d4f984e117af admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '655dca434c2647bcb0e935bfb1dc9b42', 'request_id': 'req-c0e5d4b2-2d71-422c-ae47-d4f984e117af'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:31.169633 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c0e5d4b2-2d71-422c-ae47-d4f984e117af admin admin] The token cache is being invalidate because project 655dca434c2647bcb0e935bfb1dc9b42 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:48:31.171324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c0e5d4b2-2d71-422c-ae47-d4f984e117af admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:31.173392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-49860a35-3f58-407a-a4ec-ed88f7bcd5a5 None None] MFA Rules not processed for user `f9275eeba471423e9fa2a31a6e08676d`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:31.173959 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c0e5d4b2-2d71-422c-ae47-d4f984e117af admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:31.175872 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-49860a35-3f58-407a-a4ec-ed88f7bcd5a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:31.178565 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 857/1718] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:48:30 2022] DELETE /identity/v3/projects/655dca434c2647bcb0e935bfb1dc9b42 => generated 0 bytes in 484 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:31.187797 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e5d4b2-2d71-422c-ae47-d4f984e117af admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:31.195560 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c0e5d4b2-2d71-422c-ae47-d4f984e117af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:31.318385 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 862/1719] 10.222.0.49 () {58 vars in 1009 bytes} [Fri Sep 16 03:48:31 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:31.320084 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2f89e2-a0ef-4bbe-a491-70c019924b24 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:31.320519 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2f89e2-a0ef-4bbe-a491-70c019924b24 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:31.320990 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2f89e2-a0ef-4bbe-a491-70c019924b24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:31.356866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e5d4b2-2d71-422c-ae47-d4f984e117af admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:31.375170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1c2f89e2-a0ef-4bbe-a491-70c019924b24 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:31.375170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1c2f89e2-a0ef-4bbe-a491-70c019924b24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:31.376136 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f53636-89e8-4f62-80ad-7aaa79884dc6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:31.377336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f53636-89e8-4f62-80ad-7aaa79884dc6 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:31.378427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f53636-89e8-4f62-80ad-7aaa79884dc6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:31.379902 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f53636-89e8-4f62-80ad-7aaa79884dc6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:31.383351 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 863/1720] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:31.384744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2f89e2-a0ef-4bbe-a491-70c019924b24 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:31.386091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1c2f89e2-a0ef-4bbe-a491-70c019924b24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:31.392421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-04f53636-89e8-4f62-80ad-7aaa79884dc6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:31.405905 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2f89e2-a0ef-4bbe-a491-70c019924b24 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:31.410981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5c7702-0ae3-498d-9e7e-cfc231c57825 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:31.411637 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5c7702-0ae3-498d-9e7e-cfc231c57825 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:31.411918 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5c7702-0ae3-498d-9e7e-cfc231c57825 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:31.412192 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5c7702-0ae3-498d-9e7e-cfc231c57825 admin admin] PATH_INFO: `/v3/projects/9a94dc3204094677b9ee95ab41055bf0` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:31.414845 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5c7702-0ae3-498d-9e7e-cfc231c57825 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a94dc3204094677b9ee95ab41055bf0)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:31.426574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5c7702-0ae3-498d-9e7e-cfc231c57825 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:31.431234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f53636-89e8-4f62-80ad-7aaa79884dc6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:31.433726 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f53636-89e8-4f62-80ad-7aaa79884dc6 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:31.467515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 858/1721] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 288 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:31.472388 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40894cc0-e683-4165-8460-8d2329dcc9d0 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:31.473832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40894cc0-e683-4165-8460-8d2329dcc9d0 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:31.474553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40894cc0-e683-4165-8460-8d2329dcc9d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:31.499080 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-40894cc0-e683-4165-8460-8d2329dcc9d0 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:31.500903 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-40894cc0-e683-4165-8460-8d2329dcc9d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:31.510604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 859/1722] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:31.511765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40894cc0-e683-4165-8460-8d2329dcc9d0 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:31.524877 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40894cc0-e683-4165-8460-8d2329dcc9d0 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:31.532774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe28edd-abff-4c96-b72d-2c1281bfb541 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:31.532774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe28edd-abff-4c96-b72d-2c1281bfb541 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:31.532774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe28edd-abff-4c96-b72d-2c1281bfb541 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:31.533440 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe28edd-abff-4c96-b72d-2c1281bfb541 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:31.543442 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe28edd-abff-4c96-b72d-2c1281bfb541 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:31.545692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe28edd-abff-4c96-b72d-2c1281bfb541 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:31.561919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 860/1723] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:31 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:31.566767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe28edd-abff-4c96-b72d-2c1281bfb541 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:31.580394 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe28edd-abff-4c96-b72d-2c1281bfb541 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:31.595335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9c5c7702-0ae3-498d-9e7e-cfc231c57825 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a94dc3204094677b9ee95ab41055bf0', 'request_id': 'req-9c5c7702-0ae3-498d-9e7e-cfc231c57825'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:31.615488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9c5c7702-0ae3-498d-9e7e-cfc231c57825 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a94dc3204094677b9ee95ab41055bf0', 'request_id': 'req-9c5c7702-0ae3-498d-9e7e-cfc231c57825'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:31.693540 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf339bf-46db-4691-a3b0-3285802d4c82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:31.693920 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf339bf-46db-4691-a3b0-3285802d4c82 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:31.694187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf339bf-46db-4691-a3b0-3285802d4c82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:31.694454 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf339bf-46db-4691-a3b0-3285802d4c82 admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:31.695442 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf339bf-46db-4691-a3b0-3285802d4c82 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:48:31.695442 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf339bf-46db-4691-a3b0-3285802d4c82 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:31.697572 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf339bf-46db-4691-a3b0-3285802d4c82 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:31.713134 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9c5c7702-0ae3-498d-9e7e-cfc231c57825 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a94dc3204094677b9ee95ab41055bf0', 'request_id': 'req-9c5c7702-0ae3-498d-9e7e-cfc231c57825'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:31.728478 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 861/1724] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:48:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:48:31.729398 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf339bf-46db-4691-a3b0-3285802d4c82 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:31.733518 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9c5c7702-0ae3-498d-9e7e-cfc231c57825 admin admin] The token cache is being invalidate because project 9a94dc3204094677b9ee95ab41055bf0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:48:31.733897 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9c5c7702-0ae3-498d-9e7e-cfc231c57825 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:31.739411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9c5c7702-0ae3-498d-9e7e-cfc231c57825 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:31.741517 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf339bf-46db-4691-a3b0-3285802d4c82 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:31.743814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:31.744780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 864/1725] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:48:31 2022] DELETE /identity/v3/projects/9a94dc3204094677b9ee95ab41055bf0 => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:31.748874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5c7702-0ae3-498d-9e7e-cfc231c57825 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:31.751501 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9c5c7702-0ae3-498d-9e7e-cfc231c57825 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:31.752291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d91d3ee-a8ee-4576-bcba-37a2f5391b43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:31.753188 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d91d3ee-a8ee-4576-bcba-37a2f5391b43 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:31.753564 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d91d3ee-a8ee-4576-bcba-37a2f5391b43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:31.753934 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d91d3ee-a8ee-4576-bcba-37a2f5391b43 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:31.754665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d91d3ee-a8ee-4576-bcba-37a2f5391b43 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:31.758223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d91d3ee-a8ee-4576-bcba-37a2f5391b43 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:31.775214 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5c7702-0ae3-498d-9e7e-cfc231c57825 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:31.786837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a201229-52be-40b8-890c-c85736948c91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:31.787524 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a201229-52be-40b8-890c-c85736948c91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:31.787807 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a201229-52be-40b8-890c-c85736948c91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:31.788090 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a201229-52be-40b8-890c-c85736948c91 admin admin] PATH_INFO: `/v3/users/867ea06d48104549b78442411aea7aab` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:31.789779 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a201229-52be-40b8-890c-c85736948c91 admin admin] RBAC: Authorizing `identity:delete_user(user_id=867ea06d48104549b78442411aea7aab)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:31.793338 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a201229-52be-40b8-890c-c85736948c91 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:31.933008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 862/1726] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:31 2022] POST /identity/v3/projects => generated 365 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:31.942550 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d91d3ee-a8ee-4576-bcba-37a2f5391b43 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:31.945612 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-8d91d3ee-a8ee-4576-bcba-37a2f5391b43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:32.125255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d91d3ee-a8ee-4576-bcba-37a2f5391b43 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:32.133308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4e079c-4f6c-48b2-89a9-259467198583 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:32.133717 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4e079c-4f6c-48b2-89a9-259467198583 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:32.134070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4e079c-4f6c-48b2-89a9-259467198583 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:32.134924 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4e079c-4f6c-48b2-89a9-259467198583 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:32.151888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4e079c-4f6c-48b2-89a9-259467198583 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:32.153866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4e079c-4f6c-48b2-89a9-259467198583 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:32.166543 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4a201229-52be-40b8-890c-c85736948c91 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '867ea06d48104549b78442411aea7aab'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:32.208760 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4a201229-52be-40b8-890c-c85736948c91 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '867ea06d48104549b78442411aea7aab'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:32.286187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 863/1727] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:31 2022] POST /identity/v3/users => generated 283 bytes in 349 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:32.287469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4e079c-4f6c-48b2-89a9-259467198583 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:32.304458 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4e079c-4f6c-48b2-89a9-259467198583 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:32.317037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a41a0b9-04e4-42bb-a3f1-350378ad7621 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:32.317849 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a41a0b9-04e4-42bb-a3f1-350378ad7621 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:32.318237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a41a0b9-04e4-42bb-a3f1-350378ad7621 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:32.318660 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a41a0b9-04e4-42bb-a3f1-350378ad7621 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:32.319446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a41a0b9-04e4-42bb-a3f1-350378ad7621 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:32.321609 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a41a0b9-04e4-42bb-a3f1-350378ad7621 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:32.348181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4a201229-52be-40b8-890c-c85736948c91 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '867ea06d48104549b78442411aea7aab'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:32.348900 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4a201229-52be-40b8-890c-c85736948c91 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '867ea06d48104549b78442411aea7aab'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:32.362561 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 864/1728] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:32 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:32.363948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a41a0b9-04e4-42bb-a3f1-350378ad7621 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:32.369590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4a201229-52be-40b8-890c-c85736948c91 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '867ea06d48104549b78442411aea7aab'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:32.380640 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a41a0b9-04e4-42bb-a3f1-350378ad7621 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:32.387260 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 865/1729] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:48:31 2022] DELETE /identity/v3/users/867ea06d48104549b78442411aea7aab => generated 0 bytes in 639 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:32.387503 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2aee4aa-dfa9-46df-89a3-b299002e15c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:32.388277 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2aee4aa-dfa9-46df-89a3-b299002e15c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:32.388749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2aee4aa-dfa9-46df-89a3-b299002e15c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:32.389226 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2aee4aa-dfa9-46df-89a3-b299002e15c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:32.390077 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2aee4aa-dfa9-46df-89a3-b299002e15c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:32.392210 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2aee4aa-dfa9-46df-89a3-b299002e15c0 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:32.407524 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 865/1730] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:32 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:32.418301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a201229-52be-40b8-890c-c85736948c91 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:32.419813 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4a201229-52be-40b8-890c-c85736948c91 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:32.424879 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2725479c-3b06-4f6a-b8e5-ebb78903ac15 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:32.425158 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2725479c-3b06-4f6a-b8e5-ebb78903ac15 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:32.425430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2725479c-3b06-4f6a-b8e5-ebb78903ac15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:32.465784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-2725479c-3b06-4f6a-b8e5-ebb78903ac15 None None] MFA Rules not processed for user `8f410d691e364880b13bb9144bbc5e7d`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:32.467306 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-2725479c-3b06-4f6a-b8e5-ebb78903ac15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:32.599499 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 866/1731] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:32 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:48:32.610504 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a201229-52be-40b8-890c-c85736948c91 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:32.617162 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f89b7a0-d0ff-411c-b305-87a70af1f406 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:32.618306 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f89b7a0-d0ff-411c-b305-87a70af1f406 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:32.618740 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f89b7a0-d0ff-411c-b305-87a70af1f406 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:32.619209 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f89b7a0-d0ff-411c-b305-87a70af1f406 admin admin] PATH_INFO: `/v3/projects/04f53e174db94e41bac3e45226a715e5/users/e3e9ebc6fb2c4dac8ebdf3ad170bbfd3/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:32.638207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:32.660622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f89b7a0-d0ff-411c-b305-87a70af1f406 admin admin] RBAC: Authorizing `identity:create_grant(project_id=04f53e174db94e41bac3e45226a715e5, user_id=e3e9ebc6fb2c4dac8ebdf3ad170bbfd3, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:32.664710 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f89b7a0-d0ff-411c-b305-87a70af1f406 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:32.838811 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 866/1732] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:32 2022] PUT /identity/v3/projects/04f53e174db94e41bac3e45226a715e5/users/e3e9ebc6fb2c4dac8ebdf3ad170bbfd3/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 422 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:32.853522 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3471198b-df47-4386-bd17-9dbf0111a72f None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:32.853831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3471198b-df47-4386-bd17-9dbf0111a72f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:32.854098 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3471198b-df47-4386-bd17-9dbf0111a72f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:32.884413 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d045a4-c58a-4e2b-8cf0-32791775bb45 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:32.885374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d045a4-c58a-4e2b-8cf0-32791775bb45 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:32.885654 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d045a4-c58a-4e2b-8cf0-32791775bb45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:32.930598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-3471198b-df47-4386-bd17-9dbf0111a72f None None] MFA Rules not processed for user `e3e9ebc6fb2c4dac8ebdf3ad170bbfd3`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:32.937372 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3471198b-df47-4386-bd17-9dbf0111a72f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:32.960618 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-78d045a4-c58a-4e2b-8cf0-32791775bb45 None None] MFA Rules not processed for user `a6b29b9086f64ea7a7990572567642f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:32.962273 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-78d045a4-c58a-4e2b-8cf0-32791775bb45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:33.162652 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:33.193304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 867/1733] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:32 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 341 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:33.194212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3471198b-df47-4386-bd17-9dbf0111a72f None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:33.203243 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3471198b-df47-4386-bd17-9dbf0111a72f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:33.222983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 867/1734] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:32 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 339 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:33.229543 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d045a4-c58a-4e2b-8cf0-32791775bb45 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:33.390133 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d045a4-c58a-4e2b-8cf0-32791775bb45 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:33.405925 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677fc80c-2cc1-4d66-8ce5-44d018cac8bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:33.406754 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677fc80c-2cc1-4d66-8ce5-44d018cac8bf admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:33.407077 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677fc80c-2cc1-4d66-8ce5-44d018cac8bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:33.407507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677fc80c-2cc1-4d66-8ce5-44d018cac8bf admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:33.408013 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677fc80c-2cc1-4d66-8ce5-44d018cac8bf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:33.412884 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677fc80c-2cc1-4d66-8ce5-44d018cac8bf admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:33.441999 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:33.471690 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3471198b-df47-4386-bd17-9dbf0111a72f None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:33.477044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d02425-3aa1-43da-9343-71bac1495c1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:33.477785 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d02425-3aa1-43da-9343-71bac1495c1d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:33.478013 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d02425-3aa1-43da-9343-71bac1495c1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:33.478395 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d02425-3aa1-43da-9343-71bac1495c1d admin admin] PATH_INFO: `/v3/projects/697a44c6b72645f69f92dd371730686d` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:33.479561 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d02425-3aa1-43da-9343-71bac1495c1d admin admin] RBAC: Authorizing `identity:delete_project(project_id=697a44c6b72645f69f92dd371730686d)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:33.481553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d02425-3aa1-43da-9343-71bac1495c1d admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:33.550775 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 868/1735] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:33 2022] POST /identity/v3/projects => generated 367 bytes in 327 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:33.555851 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043cfd24-df00-474f-a6e3-ad4104de75ad None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:33.555989 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043cfd24-df00-474f-a6e3-ad4104de75ad None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:33.558493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043cfd24-df00-474f-a6e3-ad4104de75ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:33.612024 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-043cfd24-df00-474f-a6e3-ad4104de75ad None None] MFA Rules not processed for user `a6b29b9086f64ea7a7990572567642f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:33.613412 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-043cfd24-df00-474f-a6e3-ad4104de75ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:33.649778 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-37d02425-3aa1-43da-9343-71bac1495c1d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '697a44c6b72645f69f92dd371730686d', 'request_id': 'req-37d02425-3aa1-43da-9343-71bac1495c1d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:33.758267 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 869/1736] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:33 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:33.759260 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043cfd24-df00-474f-a6e3-ad4104de75ad None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:33.760600 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-043cfd24-df00-474f-a6e3-ad4104de75ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:33.765549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-37d02425-3aa1-43da-9343-71bac1495c1d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '697a44c6b72645f69f92dd371730686d', 'request_id': 'req-37d02425-3aa1-43da-9343-71bac1495c1d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:33.776137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-37d02425-3aa1-43da-9343-71bac1495c1d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '697a44c6b72645f69f92dd371730686d', 'request_id': 'req-37d02425-3aa1-43da-9343-71bac1495c1d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:33.807756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-37d02425-3aa1-43da-9343-71bac1495c1d admin admin] The token cache is being invalidate because project 697a44c6b72645f69f92dd371730686d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:48:33.808056 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-37d02425-3aa1-43da-9343-71bac1495c1d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:33.813596 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-37d02425-3aa1-43da-9343-71bac1495c1d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:33.816730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 868/1737] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:48:33 2022] DELETE /identity/v3/projects/697a44c6b72645f69f92dd371730686d => generated 0 bytes in 623 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:33.817647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d02425-3aa1-43da-9343-71bac1495c1d admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:33.821157 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-37d02425-3aa1-43da-9343-71bac1495c1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:33.921296 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043cfd24-df00-474f-a6e3-ad4104de75ad None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:33.928831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7b4f6c-1228-44a8-84c3-670c06361dfe service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '176da4679fcb46148da9289fccdad5c2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:33.929478 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7b4f6c-1228-44a8-84c3-670c06361dfe service placement] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:33.930839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7b4f6c-1228-44a8-84c3-670c06361dfe service placement] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:33.930839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7b4f6c-1228-44a8-84c3-670c06361dfe service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:33.932051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8f7b4f6c-1228-44a8-84c3-670c06361dfe service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:33.944765 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7b4f6c-1228-44a8-84c3-670c06361dfe service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:33.947176 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7b4f6c-1228-44a8-84c3-670c06361dfe service placement] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:33.974969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 870/1738] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:33 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:33.976686 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7b4f6c-1228-44a8-84c3-670c06361dfe service placement] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:33.984436 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8f7b4f6c-1228-44a8-84c3-670c06361dfe service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:33.989983 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d02425-3aa1-43da-9343-71bac1495c1d admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:33.996760 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846ae988-31e2-4ee4-9bae-59d9c26b92d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:33.997692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846ae988-31e2-4ee4-9bae-59d9c26b92d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:34.005801 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846ae988-31e2-4ee4-9bae-59d9c26b92d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:34.005801 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846ae988-31e2-4ee4-9bae-59d9c26b92d1 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:34.005801 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846ae988-31e2-4ee4-9bae-59d9c26b92d1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:34.005801 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846ae988-31e2-4ee4-9bae-59d9c26b92d1 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:34.120116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 869/1739] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:33 2022] POST /identity/v3/users => generated 290 bytes in 303 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:34.125201 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846ae988-31e2-4ee4-9bae-59d9c26b92d1 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:34.132123 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-846ae988-31e2-4ee4-9bae-59d9c26b92d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:34.137638 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7b4f6c-1228-44a8-84c3-670c06361dfe service placement] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:34.149560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf47e80d-1f1a-402f-b4bc-23dc625bba42 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:34.150343 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf47e80d-1f1a-402f-b4bc-23dc625bba42 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:34.150725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf47e80d-1f1a-402f-b4bc-23dc625bba42 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:34.151165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf47e80d-1f1a-402f-b4bc-23dc625bba42 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:34.155821 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-bf47e80d-1f1a-402f-b4bc-23dc625bba42 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:34.181376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846ae988-31e2-4ee4-9bae-59d9c26b92d1 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:34.198706 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf47e80d-1f1a-402f-b4bc-23dc625bba42 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:34.203229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf47e80d-1f1a-402f-b4bc-23dc625bba42 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:34.251996 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 871/1740] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:33 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 276 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:34.253275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf47e80d-1f1a-402f-b4bc-23dc625bba42 service nova] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:34.258511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21e5f8d-8c3a-4e16-907a-0aaf21826dac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:34.260135 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21e5f8d-8c3a-4e16-907a-0aaf21826dac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:34.260627 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21e5f8d-8c3a-4e16-907a-0aaf21826dac admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:34.265697 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21e5f8d-8c3a-4e16-907a-0aaf21826dac admin admin] PATH_INFO: `/v3/users/9ce644bc53ba49aa80df3ec385a2768b` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:34.272932 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21e5f8d-8c3a-4e16-907a-0aaf21826dac admin admin] RBAC: Authorizing `identity:delete_user(user_id=9ce644bc53ba49aa80df3ec385a2768b)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:34.282971 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21e5f8d-8c3a-4e16-907a-0aaf21826dac admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:34.286219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf47e80d-1f1a-402f-b4bc-23dc625bba42 service nova] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:34.309946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a496a8e6-c4a5-454f-9d18-fd437d2b4d1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:34.326328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a496a8e6-c4a5-454f-9d18-fd437d2b4d1e admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:34.326328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a496a8e6-c4a5-454f-9d18-fd437d2b4d1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:34.333429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a496a8e6-c4a5-454f-9d18-fd437d2b4d1e admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:34.337927 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a496a8e6-c4a5-454f-9d18-fd437d2b4d1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:34.343223 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a496a8e6-c4a5-454f-9d18-fd437d2b4d1e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:34.381508 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 872/1741] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:34 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:34.386630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a496a8e6-c4a5-454f-9d18-fd437d2b4d1e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:34.410009 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a496a8e6-c4a5-454f-9d18-fd437d2b4d1e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:34.425509 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8f4abc-c9d2-4735-a96d-a98a26107166 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:34.427256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8f4abc-c9d2-4735-a96d-a98a26107166 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:34.427256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8f4abc-c9d2-4735-a96d-a98a26107166 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:34.427726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8f4abc-c9d2-4735-a96d-a98a26107166 admin admin] PATH_INFO: `/v3/projects/b3e694df24814a758811e7b2f482c808/users/432b4d98e9b54d0ebad16a3b3d854b72/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:34.462606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8f4abc-c9d2-4735-a96d-a98a26107166 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3e694df24814a758811e7b2f482c808, user_id=432b4d98e9b54d0ebad16a3b3d854b72, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:34.465759 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8f4abc-c9d2-4735-a96d-a98a26107166 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:34.591065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 873/1742] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:34 2022] PUT /identity/v3/projects/b3e694df24814a758811e7b2f482c808/users/432b4d98e9b54d0ebad16a3b3d854b72/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:34.591065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c150109-5f98-496f-9ae1-ef20c5c17f04 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:34.591065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c150109-5f98-496f-9ae1-ef20c5c17f04 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:34.591065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c150109-5f98-496f-9ae1-ef20c5c17f04 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:34.592618 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 874/1743] 10.222.0.49 () {54 vars in 912 bytes} [Fri Sep 16 03:48:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:48:34.593517 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c150109-5f98-496f-9ae1-ef20c5c17f04 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:34.680054 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c150109-5f98-496f-9ae1-ef20c5c17f04 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:34.688572 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438ef6a5-daf9-4251-8765-badf336675ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:34.689660 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438ef6a5-daf9-4251-8765-badf336675ac admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:34.690217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438ef6a5-daf9-4251-8765-badf336675ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:34.690808 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438ef6a5-daf9-4251-8765-badf336675ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:34.691759 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438ef6a5-daf9-4251-8765-badf336675ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:34.694806 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438ef6a5-daf9-4251-8765-badf336675ac admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:34.708473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 875/1744] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:34 2022] GET /identity/v3/roles => generated 1420 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:34.714126 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20edb63d-da9b-4194-9e96-4071a03c9971 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:34.714573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20edb63d-da9b-4194-9e96-4071a03c9971 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:34.714995 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20edb63d-da9b-4194-9e96-4071a03c9971 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:34.742696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-20edb63d-da9b-4194-9e96-4071a03c9971 None None] MFA Rules not processed for user `f9275eeba471423e9fa2a31a6e08676d`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:34.746726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-20edb63d-da9b-4194-9e96-4071a03c9971 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:34.754357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a21e5f8d-8c3a-4e16-907a-0aaf21826dac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9ce644bc53ba49aa80df3ec385a2768b'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:34.853908 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a21e5f8d-8c3a-4e16-907a-0aaf21826dac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9ce644bc53ba49aa80df3ec385a2768b'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:34.884639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a21e5f8d-8c3a-4e16-907a-0aaf21826dac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9ce644bc53ba49aa80df3ec385a2768b'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:34.885608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a21e5f8d-8c3a-4e16-907a-0aaf21826dac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9ce644bc53ba49aa80df3ec385a2768b'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:34.896742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 876/1745] 10.222.0.49 () {58 vars in 1009 bytes} [Fri Sep 16 03:48:34 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:34.897883 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20edb63d-da9b-4194-9e96-4071a03c9971 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:34.901528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-20edb63d-da9b-4194-9e96-4071a03c9971 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:34.907617 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a21e5f8d-8c3a-4e16-907a-0aaf21826dac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9ce644bc53ba49aa80df3ec385a2768b'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:34.921698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 870/1746] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:48:34 2022] DELETE /identity/v3/users/9ce644bc53ba49aa80df3ec385a2768b => generated 0 bytes in 798 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:34.926986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21e5f8d-8c3a-4e16-907a-0aaf21826dac admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:34.933787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a21e5f8d-8c3a-4e16-907a-0aaf21826dac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:34.935409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20edb63d-da9b-4194-9e96-4071a03c9971 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:35.125153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21e5f8d-8c3a-4e16-907a-0aaf21826dac admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:35.126778 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fc4d09-1858-483b-b3ef-c71f4b78e84f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:35.127813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a1bca6-8434-47ee-9d43-5dc2d66be837 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:35.128282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fc4d09-1858-483b-b3ef-c71f4b78e84f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:35.128282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fc4d09-1858-483b-b3ef-c71f4b78e84f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:35.128698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a1bca6-8434-47ee-9d43-5dc2d66be837 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:35.129147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fc4d09-1858-483b-b3ef-c71f4b78e84f admin admin] PATH_INFO: `/v3/projects/ef431b2d03774380a4ead2dd4ff575c7` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:35.129439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a1bca6-8434-47ee-9d43-5dc2d66be837 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:35.129439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a1bca6-8434-47ee-9d43-5dc2d66be837 admin admin] PATH_INFO: `/v3/projects/b3e694df24814a758811e7b2f482c808/users/432b4d98e9b54d0ebad16a3b3d854b72/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:35.129725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fc4d09-1858-483b-b3ef-c71f4b78e84f admin admin] RBAC: Authorizing `identity:delete_project(project_id=ef431b2d03774380a4ead2dd4ff575c7)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:35.129725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fc4d09-1858-483b-b3ef-c71f4b78e84f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:35.129893 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a1bca6-8434-47ee-9d43-5dc2d66be837 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3e694df24814a758811e7b2f482c808, user_id=432b4d98e9b54d0ebad16a3b3d854b72, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:35.129893 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a1bca6-8434-47ee-9d43-5dc2d66be837 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:35.165437 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 871/1747] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:34 2022] PUT /identity/v3/projects/b3e694df24814a758811e7b2f482c808/users/432b4d98e9b54d0ebad16a3b3d854b72/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:35.166865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a1bca6-8434-47ee-9d43-5dc2d66be837 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:35.173535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-71a1bca6-8434-47ee-9d43-5dc2d66be837 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:35.428178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a1bca6-8434-47ee-9d43-5dc2d66be837 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:35.437200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18087967-fca1-4b33-880c-88d9127e981f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:35.438211 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18087967-fca1-4b33-880c-88d9127e981f service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:35.438647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18087967-fca1-4b33-880c-88d9127e981f service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:35.439096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18087967-fca1-4b33-880c-88d9127e981f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:35.444744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-18087967-fca1-4b33-880c-88d9127e981f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:35.468886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18087967-fca1-4b33-880c-88d9127e981f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:35.471929 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18087967-fca1-4b33-880c-88d9127e981f service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:35.484118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 872/1748] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:35 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:35.486482 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57ab490-74ef-4fb8-a412-01911bbf2d6f None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:35.487578 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57ab490-74ef-4fb8-a412-01911bbf2d6f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:35.488519 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57ab490-74ef-4fb8-a412-01911bbf2d6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:35.545744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-b57ab490-74ef-4fb8-a412-01911bbf2d6f None None] MFA Rules not processed for user `432b4d98e9b54d0ebad16a3b3d854b72`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:35.548171 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b57ab490-74ef-4fb8-a412-01911bbf2d6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:35.658104 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 873/1749] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:35 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:35.673350 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57ab490-74ef-4fb8-a412-01911bbf2d6f None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:35.675141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b57ab490-74ef-4fb8-a412-01911bbf2d6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:35.751509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57ab490-74ef-4fb8-a412-01911bbf2d6f None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:35.755649 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576cbb6c-c71e-4010-83ae-7732745b2088 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:35.756901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576cbb6c-c71e-4010-83ae-7732745b2088 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:35.757037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576cbb6c-c71e-4010-83ae-7732745b2088 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:35.757243 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576cbb6c-c71e-4010-83ae-7732745b2088 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:35.770766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576cbb6c-c71e-4010-83ae-7732745b2088 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:35.772263 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576cbb6c-c71e-4010-83ae-7732745b2088 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:35.788221 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 874/1750] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:35 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:37.172316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-54fc4d09-1858-483b-b3ef-c71f4b78e84f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ef431b2d03774380a4ead2dd4ff575c7', 'request_id': 'req-54fc4d09-1858-483b-b3ef-c71f4b78e84f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:37.179133 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-54fc4d09-1858-483b-b3ef-c71f4b78e84f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ef431b2d03774380a4ead2dd4ff575c7', 'request_id': 'req-54fc4d09-1858-483b-b3ef-c71f4b78e84f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:37.318636 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-54fc4d09-1858-483b-b3ef-c71f4b78e84f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ef431b2d03774380a4ead2dd4ff575c7', 'request_id': 'req-54fc4d09-1858-483b-b3ef-c71f4b78e84f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:37.340174 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-54fc4d09-1858-483b-b3ef-c71f4b78e84f admin admin] The token cache is being invalidate because project ef431b2d03774380a4ead2dd4ff575c7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:48:37.340488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-54fc4d09-1858-483b-b3ef-c71f4b78e84f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:37.344560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-54fc4d09-1858-483b-b3ef-c71f4b78e84f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:37.349269 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 877/1751] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:48:34 2022] DELETE /identity/v3/projects/ef431b2d03774380a4ead2dd4ff575c7 => generated 0 bytes in 2452 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:37.355021 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b8faaa-b34c-4f7a-8d47-fc8cb85c2ab2 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:37.355021 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b8faaa-b34c-4f7a-8d47-fc8cb85c2ab2 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:37.355021 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b8faaa-b34c-4f7a-8d47-fc8cb85c2ab2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:37.427571 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fc4d09-1858-483b-b3ef-c71f4b78e84f admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:37.429248 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-54fc4d09-1858-483b-b3ef-c71f4b78e84f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:37.451775 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-01b8faaa-b34c-4f7a-8d47-fc8cb85c2ab2 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:37.454659 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-01b8faaa-b34c-4f7a-8d47-fc8cb85c2ab2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:37.695963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fc4d09-1858-483b-b3ef-c71f4b78e84f admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:37.707626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83085841-f4fd-405d-89f1-4447ab8ee789 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:37.712512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83085841-f4fd-405d-89f1-4447ab8ee789 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:37.715590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83085841-f4fd-405d-89f1-4447ab8ee789 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:37.715898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83085841-f4fd-405d-89f1-4447ab8ee789 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:37.728771 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-83085841-f4fd-405d-89f1-4447ab8ee789 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:37.734631 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 875/1752] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 381 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:37.736190 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472cf619-2609-45c5-bc76-c3339fa1be18 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:37.736478 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472cf619-2609-45c5-bc76-c3339fa1be18 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:37.736805 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472cf619-2609-45c5-bc76-c3339fa1be18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:37.802604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-472cf619-2609-45c5-bc76-c3339fa1be18 None None] MFA Rules not processed for user `ff7df9be8475405eba924b59a7bd6bb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:37.818516 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-472cf619-2609-45c5-bc76-c3339fa1be18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:37.822713 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:37.950266 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83085841-f4fd-405d-89f1-4447ab8ee789 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:37.953011 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83085841-f4fd-405d-89f1-4447ab8ee789 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:37.991864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 878/1753] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:37 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 565 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:37.994594 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 876/1754] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:37 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:37.997307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e85b5c-54ec-4ac1-8086-d0f9ece631f7 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:37.997750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e85b5c-54ec-4ac1-8086-d0f9ece631f7 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:38.009926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91327300-bad7-4681-b382-208abef2346f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:38.010239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91327300-bad7-4681-b382-208abef2346f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:38.012908 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91327300-bad7-4681-b382-208abef2346f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:38.014825 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e85b5c-54ec-4ac1-8086-d0f9ece631f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:38.093048 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-b0e85b5c-54ec-4ac1-8086-d0f9ece631f7 None None] MFA Rules not processed for user `a6b29b9086f64ea7a7990572567642f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:38.093338 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-91327300-bad7-4681-b382-208abef2346f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:38.094865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b0e85b5c-54ec-4ac1-8086-d0f9ece631f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:38.098317 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-91327300-bad7-4681-b382-208abef2346f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:38.142553 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 879/1755] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:38.146837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91327300-bad7-4681-b382-208abef2346f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:38.174441 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91327300-bad7-4681-b382-208abef2346f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:38.191462 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcaa71b8-2df1-4960-89c6-71203cac4407 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:38.202517 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcaa71b8-2df1-4960-89c6-71203cac4407 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:38.209593 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcaa71b8-2df1-4960-89c6-71203cac4407 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:38.212823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcaa71b8-2df1-4960-89c6-71203cac4407 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:38.219641 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcaa71b8-2df1-4960-89c6-71203cac4407 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:48:38.220056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcaa71b8-2df1-4960-89c6-71203cac4407 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:38.224695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:38.234983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcaa71b8-2df1-4960-89c6-71203cac4407 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:38.242885 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 877/1756] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:37 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 248 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:38.246686 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e85b5c-54ec-4ac1-8086-d0f9ece631f7 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:38.250378 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b0e85b5c-54ec-4ac1-8086-d0f9ece631f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:38.270078 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 880/1757] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:48:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:48:38.274596 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcaa71b8-2df1-4960-89c6-71203cac4407 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:38.278999 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e85b5c-54ec-4ac1-8086-d0f9ece631f7 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:38.300486 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcaa71b8-2df1-4960-89c6-71203cac4407 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:38.301532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a9f598f-87a4-4c52-bec1-7c5383d503cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:38.302267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9f598f-87a4-4c52-bec1-7c5383d503cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:38.302730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9f598f-87a4-4c52-bec1-7c5383d503cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:38.303126 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9f598f-87a4-4c52-bec1-7c5383d503cc admin admin] PATH_INFO: `/v3/projects/479b376d470947e2a284f08e804ca174` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:38.305493 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a9f598f-87a4-4c52-bec1-7c5383d503cc admin admin] RBAC: Authorizing `identity:delete_project(project_id=479b376d470947e2a284f08e804ca174)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:38.309053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a9f598f-87a4-4c52-bec1-7c5383d503cc admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:38.313739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b59759-37b4-4536-a83d-823a63c6cfb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:38.315936 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b59759-37b4-4536-a83d-823a63c6cfb3 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:38.317336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b59759-37b4-4536-a83d-823a63c6cfb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:38.320738 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b59759-37b4-4536-a83d-823a63c6cfb3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:38.322659 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b59759-37b4-4536-a83d-823a63c6cfb3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:38.336850 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b59759-37b4-4536-a83d-823a63c6cfb3 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:38.509731 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 881/1758] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:38 2022] POST /identity/v3/projects => generated 385 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:38.511512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b59759-37b4-4536-a83d-823a63c6cfb3 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:38.632161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1a9f598f-87a4-4c52-bec1-7c5383d503cc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '479b376d470947e2a284f08e804ca174', 'request_id': 'req-1a9f598f-87a4-4c52-bec1-7c5383d503cc'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:38.768895 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1a9f598f-87a4-4c52-bec1-7c5383d503cc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '479b376d470947e2a284f08e804ca174', 'request_id': 'req-1a9f598f-87a4-4c52-bec1-7c5383d503cc'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:38.782500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b59759-37b4-4536-a83d-823a63c6cfb3 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:38.787685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1a9f598f-87a4-4c52-bec1-7c5383d503cc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '479b376d470947e2a284f08e804ca174', 'request_id': 'req-1a9f598f-87a4-4c52-bec1-7c5383d503cc'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:38.830020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1a9f598f-87a4-4c52-bec1-7c5383d503cc admin admin] The token cache is being invalidate because project 479b376d470947e2a284f08e804ca174 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:48:38.833360 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1a9f598f-87a4-4c52-bec1-7c5383d503cc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:38.834754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1a9f598f-87a4-4c52-bec1-7c5383d503cc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:38.841088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 878/1759] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:48:38 2022] DELETE /identity/v3/projects/479b376d470947e2a284f08e804ca174 => generated 0 bytes in 597 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:38.848191 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15530925-9294-44dc-94a4-98e1b9ee96cf None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:38.848478 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15530925-9294-44dc-94a4-98e1b9ee96cf None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:38.851690 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15530925-9294-44dc-94a4-98e1b9ee96cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:38.933685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-15530925-9294-44dc-94a4-98e1b9ee96cf None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:38.955775 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-15530925-9294-44dc-94a4-98e1b9ee96cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:38.986499 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4716cc8f-d564-4584-a349-890c7ae163c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:38.989687 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4716cc8f-d564-4584-a349-890c7ae163c0 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:38.990107 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4716cc8f-d564-4584-a349-890c7ae163c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:38.990491 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4716cc8f-d564-4584-a349-890c7ae163c0 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:38.991593 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4716cc8f-d564-4584-a349-890c7ae163c0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:38.997773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4716cc8f-d564-4584-a349-890c7ae163c0 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:39.142389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 882/1760] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:38 2022] POST /identity/v3/users => generated 293 bytes in 632 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:48:39.143563 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4716cc8f-d564-4584-a349-890c7ae163c0 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:39.148040 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4716cc8f-d564-4584-a349-890c7ae163c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:39.178508 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4716cc8f-d564-4584-a349-890c7ae163c0 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:39.184573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 879/1761] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 338 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:39.194189 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d67c611-47e9-42aa-8381-19799c386514 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:39.194493 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d67c611-47e9-42aa-8381-19799c386514 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:39.195552 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d67c611-47e9-42aa-8381-19799c386514 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:39.199663 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebe83a3-d4b5-45d0-bf57-76beeabbab88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:39.200417 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebe83a3-d4b5-45d0-bf57-76beeabbab88 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:39.200796 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebe83a3-d4b5-45d0-bf57-76beeabbab88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:39.201135 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebe83a3-d4b5-45d0-bf57-76beeabbab88 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:39.206373 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebe83a3-d4b5-45d0-bf57-76beeabbab88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:39.208331 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebe83a3-d4b5-45d0-bf57-76beeabbab88 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:39.238317 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 883/1762] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:39 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:39.241973 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebe83a3-d4b5-45d0-bf57-76beeabbab88 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:39.268755 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebe83a3-d4b5-45d0-bf57-76beeabbab88 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:39.277736 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-5d67c611-47e9-42aa-8381-19799c386514 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:39.280146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-5d67c611-47e9-42aa-8381-19799c386514 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:39.285899 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4bd4ba-2973-4080-9ac7-0e3275ebf5c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:39.290317 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4bd4ba-2973-4080-9ac7-0e3275ebf5c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:39.290704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4bd4ba-2973-4080-9ac7-0e3275ebf5c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:39.291109 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4bd4ba-2973-4080-9ac7-0e3275ebf5c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:39.294028 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4bd4ba-2973-4080-9ac7-0e3275ebf5c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:39.300019 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4bd4ba-2973-4080-9ac7-0e3275ebf5c0 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:39.320932 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 880/1763] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:39.323807 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d67c611-47e9-42aa-8381-19799c386514 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:39.346960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 884/1764] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:39 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:39.350177 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d67c611-47e9-42aa-8381-19799c386514 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:39.356196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4bd4ba-2973-4080-9ac7-0e3275ebf5c0 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:39.367714 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cf50e8-6b5f-4fbd-b6c3-b2b489f81543 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:39.369685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cf50e8-6b5f-4fbd-b6c3-b2b489f81543 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:39.371522 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cf50e8-6b5f-4fbd-b6c3-b2b489f81543 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:39.373051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cf50e8-6b5f-4fbd-b6c3-b2b489f81543 admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:39.374218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cf50e8-6b5f-4fbd-b6c3-b2b489f81543 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:48:39.377929 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cf50e8-6b5f-4fbd-b6c3-b2b489f81543 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:39.381899 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cf50e8-6b5f-4fbd-b6c3-b2b489f81543 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:39.393689 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4bd4ba-2973-4080-9ac7-0e3275ebf5c0 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:39.415690 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0b203d-64aa-4d9e-8ccc-728c5a5d621d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:39.419044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0b203d-64aa-4d9e-8ccc-728c5a5d621d admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:39.422414 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0b203d-64aa-4d9e-8ccc-728c5a5d621d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:39.422414 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0b203d-64aa-4d9e-8ccc-728c5a5d621d admin admin] PATH_INFO: `/v3/projects/7bcc9b90fae04761a6382184e7381df4/users/ecf17cdcc82d4c1e9376db42e1a20b1e/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:39.431094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 881/1765] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:48:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:48:39.433075 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cf50e8-6b5f-4fbd-b6c3-b2b489f81543 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:39.470404 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cf50e8-6b5f-4fbd-b6c3-b2b489f81543 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:39.478810 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0b203d-64aa-4d9e-8ccc-728c5a5d621d admin admin] RBAC: Authorizing `identity:create_grant(project_id=7bcc9b90fae04761a6382184e7381df4, user_id=ecf17cdcc82d4c1e9376db42e1a20b1e, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:39.481474 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edb9f82-a166-4514-8108-008d086385b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:39.482110 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edb9f82-a166-4514-8108-008d086385b7 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:39.482413 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edb9f82-a166-4514-8108-008d086385b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:39.483728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edb9f82-a166-4514-8108-008d086385b7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:39.487256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0b203d-64aa-4d9e-8ccc-728c5a5d621d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:39.489885 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6edb9f82-a166-4514-8108-008d086385b7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:39.492123 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6edb9f82-a166-4514-8108-008d086385b7 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:39.688699 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 885/1766] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:39 2022] PUT /identity/v3/projects/7bcc9b90fae04761a6382184e7381df4/users/ecf17cdcc82d4c1e9376db42e1a20b1e/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:39.698876 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3227d684-40a8-4651-b4fc-a69d1fc0421d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:39.702299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3227d684-40a8-4651-b4fc-a69d1fc0421d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:39.702987 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3227d684-40a8-4651-b4fc-a69d1fc0421d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:39.706672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 882/1767] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:39 2022] POST /identity/v3/projects => generated 401 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:39.721141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0ccada-b353-4653-b08e-ded2b65a1eda None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:39.722013 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0ccada-b353-4653-b08e-ded2b65a1eda None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:39.722498 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0ccada-b353-4653-b08e-ded2b65a1eda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:39.764648 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-3227d684-40a8-4651-b4fc-a69d1fc0421d None None] MFA Rules not processed for user `416479ba056f4612b6214cf115251269`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:39.765967 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3227d684-40a8-4651-b4fc-a69d1fc0421d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:39.792098 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-ee0ccada-b353-4653-b08e-ded2b65a1eda None None] MFA Rules not processed for user `ecf17cdcc82d4c1e9376db42e1a20b1e`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:39.793597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ee0ccada-b353-4653-b08e-ded2b65a1eda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:39.870610 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 886/1768] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:39 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:39.870610 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3227d684-40a8-4651-b4fc-a69d1fc0421d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:39.938714 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 883/1769] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:39 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:39.942823 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0ccada-b353-4653-b08e-ded2b65a1eda None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:39.997721 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3227d684-40a8-4651-b4fc-a69d1fc0421d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:40.005223 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d92c7fa-3020-4a99-be31-18910d04ebcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:40.006174 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d92c7fa-3020-4a99-be31-18910d04ebcd admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:40.006703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d92c7fa-3020-4a99-be31-18910d04ebcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:40.007325 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d92c7fa-3020-4a99-be31-18910d04ebcd admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:40.008276 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d92c7fa-3020-4a99-be31-18910d04ebcd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:40.018644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d92c7fa-3020-4a99-be31-18910d04ebcd admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:40.104136 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0ccada-b353-4653-b08e-ded2b65a1eda None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:40.126561 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efcd5132-0025-4bc2-a15a-5b9a1dfbe975 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:40.127625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcd5132-0025-4bc2-a15a-5b9a1dfbe975 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:40.127783 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcd5132-0025-4bc2-a15a-5b9a1dfbe975 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:40.128177 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcd5132-0025-4bc2-a15a-5b9a1dfbe975 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:40.128980 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efcd5132-0025-4bc2-a15a-5b9a1dfbe975 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:40.131880 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efcd5132-0025-4bc2-a15a-5b9a1dfbe975 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:40.219073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 887/1770] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:39 2022] POST /identity/v3/users => generated 301 bytes in 343 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:40.219073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d92c7fa-3020-4a99-be31-18910d04ebcd admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:40.280877 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d92c7fa-3020-4a99-be31-18910d04ebcd admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:40.322597 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac4e3a3-140a-4dac-b9ab-ae1b38094191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:40.325972 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac4e3a3-140a-4dac-b9ab-ae1b38094191 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:40.332525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac4e3a3-140a-4dac-b9ab-ae1b38094191 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:40.332525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac4e3a3-140a-4dac-b9ab-ae1b38094191 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:40.333126 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac4e3a3-140a-4dac-b9ab-ae1b38094191 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:40.334769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac4e3a3-140a-4dac-b9ab-ae1b38094191 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:40.346513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 884/1771] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:39 2022] POST /identity/v3/projects => generated 387 bytes in 404 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:48:40.354382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efcd5132-0025-4bc2-a15a-5b9a1dfbe975 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:40.450665 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 888/1772] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:40 2022] GET /identity/v3/roles => generated 1420 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:40.452113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac4e3a3-140a-4dac-b9ab-ae1b38094191 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:40.474559 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=88,keystone:DELETE=8,keystone:INSERT=4 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:40.475324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac4e3a3-140a-4dac-b9ab-ae1b38094191 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:40.668821 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efcd5132-0025-4bc2-a15a-5b9a1dfbe975 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:40.711779 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be865f6b-74bc-4065-b768-4ce6dfc151f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:40.711779 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be865f6b-74bc-4065-b768-4ce6dfc151f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:40.711779 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be865f6b-74bc-4065-b768-4ce6dfc151f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:40.711779 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be865f6b-74bc-4065-b768-4ce6dfc151f8 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:40.712704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be865f6b-74bc-4065-b768-4ce6dfc151f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:40.712704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be865f6b-74bc-4065-b768-4ce6dfc151f8 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:40.774203 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a356bb-b004-4d95-9909-6e8447491fd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:40.776426 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a356bb-b004-4d95-9909-6e8447491fd2 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:40.776672 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a356bb-b004-4d95-9909-6e8447491fd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:40.778538 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a356bb-b004-4d95-9909-6e8447491fd2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:40.782102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a356bb-b004-4d95-9909-6e8447491fd2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:40.786712 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a356bb-b004-4d95-9909-6e8447491fd2 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:40.813776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 889/1773] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:40 2022] GET /identity/v3/roles => generated 1420 bytes in 363 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:40.818729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a356bb-b004-4d95-9909-6e8447491fd2 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:40.850800 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a356bb-b004-4d95-9909-6e8447491fd2 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:40.871374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31530e5b-5a73-4c0f-9ac1-1822e332ccd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:40.872071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31530e5b-5a73-4c0f-9ac1-1822e332ccd2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:40.872360 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31530e5b-5a73-4c0f-9ac1-1822e332ccd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:40.872638 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31530e5b-5a73-4c0f-9ac1-1822e332ccd2 admin admin] PATH_INFO: `/v3/projects/7f7b6c5700b3450b8cc88efe291af9c7/users/2b35fa35fcf140ec8299992b60dca043/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:40.916479 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31530e5b-5a73-4c0f-9ac1-1822e332ccd2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f7b6c5700b3450b8cc88efe291af9c7, user_id=2b35fa35fcf140ec8299992b60dca043, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:40.919142 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31530e5b-5a73-4c0f-9ac1-1822e332ccd2 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:41.015873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 885/1774] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:40 2022] POST /identity/v3/users => generated 300 bytes in 662 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:41.016849 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be865f6b-74bc-4065-b768-4ce6dfc151f8 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:41.040587 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 890/1775] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:40 2022] PUT /identity/v3/projects/7f7b6c5700b3450b8cc88efe291af9c7/users/2b35fa35fcf140ec8299992b60dca043/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:48:41.048681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ced18e-a814-4318-9048-3953881ae6bf None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:41.048979 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ced18e-a814-4318-9048-3953881ae6bf None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:41.049259 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ced18e-a814-4318-9048-3953881ae6bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:41.057602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be865f6b-74bc-4065-b768-4ce6dfc151f8 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:41.115819 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a4ced18e-a814-4318-9048-3953881ae6bf None None] MFA Rules not processed for user `2b35fa35fcf140ec8299992b60dca043`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:41.117177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a4ced18e-a814-4318-9048-3953881ae6bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:41.170634 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c88a07a-35e5-43c9-8f13-ced5fbd48571 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:41.171558 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c88a07a-35e5-43c9-8f13-ced5fbd48571 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:41.171944 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c88a07a-35e5-43c9-8f13-ced5fbd48571 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:41.172285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c88a07a-35e5-43c9-8f13-ced5fbd48571 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:41.173035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c88a07a-35e5-43c9-8f13-ced5fbd48571 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:41.174963 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c88a07a-35e5-43c9-8f13-ced5fbd48571 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:41.189969 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 886/1776] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:41 2022] GET /identity/v3/roles => generated 1420 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:41.191100 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c88a07a-35e5-43c9-8f13-ced5fbd48571 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:41.204078 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c88a07a-35e5-43c9-8f13-ced5fbd48571 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:41.209097 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9d2743-eeba-45c5-a4cb-a00af08fe637 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:41.220111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9d2743-eeba-45c5-a4cb-a00af08fe637 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:41.220401 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9d2743-eeba-45c5-a4cb-a00af08fe637 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:41.222524 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9d2743-eeba-45c5-a4cb-a00af08fe637 admin admin] PATH_INFO: `/v3/projects/768e634e5f60449eb49465798f8c167a/users/7843f2dc7efa453e9f65b8233d3e1419/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:41.242146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:41.242146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 891/1777] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:41 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:41.245848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ced18e-a814-4318-9048-3953881ae6bf None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:41.266305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9d2743-eeba-45c5-a4cb-a00af08fe637 admin admin] RBAC: Authorizing `identity:create_grant(project_id=768e634e5f60449eb49465798f8c167a, user_id=7843f2dc7efa453e9f65b8233d3e1419, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:41.268552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ced18e-a814-4318-9048-3953881ae6bf None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:41.269631 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9d2743-eeba-45c5-a4cb-a00af08fe637 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:41.274822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cddbfdb-7713-470e-b125-fccc88a2d797 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:41.276713 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cddbfdb-7713-470e-b125-fccc88a2d797 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:41.277217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cddbfdb-7713-470e-b125-fccc88a2d797 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:41.277685 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cddbfdb-7713-470e-b125-fccc88a2d797 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:41.279181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cddbfdb-7713-470e-b125-fccc88a2d797 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:41.282599 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cddbfdb-7713-470e-b125-fccc88a2d797 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:41.424976 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 887/1778] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:41 2022] PUT /identity/v3/projects/768e634e5f60449eb49465798f8c167a/users/7843f2dc7efa453e9f65b8233d3e1419/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:41.433073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9d2743-eeba-45c5-a4cb-a00af08fe637 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:41.484243 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 892/1779] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:41 2022] POST /identity/v3/projects => generated 399 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:41.486234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cddbfdb-7713-470e-b125-fccc88a2d797 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:41.648853 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9d2743-eeba-45c5-a4cb-a00af08fe637 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:41.709735 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cddbfdb-7713-470e-b125-fccc88a2d797 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:41.718734 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9651fc44-fcd7-44b7-bb4b-9476b1c17841 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:41.720115 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9651fc44-fcd7-44b7-bb4b-9476b1c17841 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:41.720409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9651fc44-fcd7-44b7-bb4b-9476b1c17841 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:41.721006 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9651fc44-fcd7-44b7-bb4b-9476b1c17841 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:41.721658 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9651fc44-fcd7-44b7-bb4b-9476b1c17841 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:41.741371 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9651fc44-fcd7-44b7-bb4b-9476b1c17841 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:41.826832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656230a4-2d90-4a3b-9a1d-5d80cce4ce24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:41.833329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656230a4-2d90-4a3b-9a1d-5d80cce4ce24 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:41.833329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656230a4-2d90-4a3b-9a1d-5d80cce4ce24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:41.833329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656230a4-2d90-4a3b-9a1d-5d80cce4ce24 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:41.833329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656230a4-2d90-4a3b-9a1d-5d80cce4ce24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:41.833329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656230a4-2d90-4a3b-9a1d-5d80cce4ce24 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:41.878959 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 888/1780] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:41 2022] GET /identity/v3/roles => generated 1420 bytes in 447 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:41.883474 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656230a4-2d90-4a3b-9a1d-5d80cce4ce24 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:41.897962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 893/1781] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:41 2022] POST /identity/v3/users => generated 306 bytes in 412 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:41.908074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656230a4-2d90-4a3b-9a1d-5d80cce4ce24 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:41.909286 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9651fc44-fcd7-44b7-bb4b-9476b1c17841 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:41.930660 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55abef7b-a3ac-4c3a-85d9-bbcb0871e369 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:41.930660 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55abef7b-a3ac-4c3a-85d9-bbcb0871e369 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:41.930660 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55abef7b-a3ac-4c3a-85d9-bbcb0871e369 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:41.930660 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55abef7b-a3ac-4c3a-85d9-bbcb0871e369 admin admin] PATH_INFO: `/v3/projects/768e634e5f60449eb49465798f8c167a/users/7843f2dc7efa453e9f65b8233d3e1419/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:41.933404 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55abef7b-a3ac-4c3a-85d9-bbcb0871e369 admin admin] RBAC: Authorizing `identity:create_grant(project_id=768e634e5f60449eb49465798f8c167a, user_id=7843f2dc7efa453e9f65b8233d3e1419, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:41.945535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55abef7b-a3ac-4c3a-85d9-bbcb0871e369 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:41.956012 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9651fc44-fcd7-44b7-bb4b-9476b1c17841 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:41.966161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 889/1782] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:41 2022] PUT /identity/v3/projects/768e634e5f60449eb49465798f8c167a/users/7843f2dc7efa453e9f65b8233d3e1419/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:48:41.979231 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0f07e3-309e-42e1-a910-2c1c5797a117 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:41.979759 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0f07e3-309e-42e1-a910-2c1c5797a117 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:41.980329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0f07e3-309e-42e1-a910-2c1c5797a117 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:42.075833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b594c31-f3b3-4d92-88ed-fdb7d8458d21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:42.076547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b594c31-f3b3-4d92-88ed-fdb7d8458d21 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:42.076835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b594c31-f3b3-4d92-88ed-fdb7d8458d21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:42.079138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b594c31-f3b3-4d92-88ed-fdb7d8458d21 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:42.085439 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b594c31-f3b3-4d92-88ed-fdb7d8458d21 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:42.091578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b594c31-f3b3-4d92-88ed-fdb7d8458d21 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:42.109226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 894/1783] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:41 2022] GET /identity/v3/roles => generated 1420 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:42.112843 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b594c31-f3b3-4d92-88ed-fdb7d8458d21 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:42.143357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b594c31-f3b3-4d92-88ed-fdb7d8458d21 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:42.172140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-dd0f07e3-309e-42e1-a910-2c1c5797a117 None None] MFA Rules not processed for user `7843f2dc7efa453e9f65b8233d3e1419`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:42.175585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-dd0f07e3-309e-42e1-a910-2c1c5797a117 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:42.204379 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40aa102e-d7ad-4988-be65-81d5d30a7caf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:42.205454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aa102e-d7ad-4988-be65-81d5d30a7caf admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:42.205897 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aa102e-d7ad-4988-be65-81d5d30a7caf admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:42.206447 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aa102e-d7ad-4988-be65-81d5d30a7caf admin admin] PATH_INFO: `/v3/projects/3002c2f1222949079ed30d3e444c6405/users/dcefd7a94ef2407b875e112b37e8116e/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:42.257383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40aa102e-d7ad-4988-be65-81d5d30a7caf admin admin] RBAC: Authorizing `identity:create_grant(project_id=3002c2f1222949079ed30d3e444c6405, user_id=dcefd7a94ef2407b875e112b37e8116e, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:42.276187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40aa102e-d7ad-4988-be65-81d5d30a7caf admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:42.310191 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 890/1784] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:41 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 329 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:42.320275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0f07e3-309e-42e1-a910-2c1c5797a117 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:42.322573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-dd0f07e3-309e-42e1-a910-2c1c5797a117 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:42.469411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 895/1785] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:42 2022] PUT /identity/v3/projects/3002c2f1222949079ed30d3e444c6405/users/dcefd7a94ef2407b875e112b37e8116e/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:42.470756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40aa102e-d7ad-4988-be65-81d5d30a7caf admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:42.473198 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0f07e3-309e-42e1-a910-2c1c5797a117 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:42.561220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd6b6ee-4f6a-4398-8e69-4d33676f6de9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:42.562030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd6b6ee-4f6a-4398-8e69-4d33676f6de9 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:42.563150 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd6b6ee-4f6a-4398-8e69-4d33676f6de9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:42.563150 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd6b6ee-4f6a-4398-8e69-4d33676f6de9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:42.612447 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40aa102e-d7ad-4988-be65-81d5d30a7caf admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:42.618507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce8b9b1-1c47-4783-98d4-248cd0037200 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:42.619301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce8b9b1-1c47-4783-98d4-248cd0037200 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:42.619698 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce8b9b1-1c47-4783-98d4-248cd0037200 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:42.620064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce8b9b1-1c47-4783-98d4-248cd0037200 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:42.620884 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce8b9b1-1c47-4783-98d4-248cd0037200 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:42.622636 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce8b9b1-1c47-4783-98d4-248cd0037200 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:42.634624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 896/1786] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:42 2022] GET /identity/v3/roles => generated 1420 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:42.639160 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce8b9b1-1c47-4783-98d4-248cd0037200 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:42.648442 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd6b6ee-4f6a-4398-8e69-4d33676f6de9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:42.649910 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd6b6ee-4f6a-4398-8e69-4d33676f6de9 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:42.651448 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce8b9b1-1c47-4783-98d4-248cd0037200 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:42.658840 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713494d7-cab8-49c0-aa11-9fbb57da1a3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:42.659801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713494d7-cab8-49c0-aa11-9fbb57da1a3d admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:42.660280 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 891/1787] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:42 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:48:42.660796 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713494d7-cab8-49c0-aa11-9fbb57da1a3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:42.661093 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713494d7-cab8-49c0-aa11-9fbb57da1a3d admin admin] PATH_INFO: `/v3/projects/3002c2f1222949079ed30d3e444c6405/users/dcefd7a94ef2407b875e112b37e8116e/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:42.663402 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713494d7-cab8-49c0-aa11-9fbb57da1a3d admin admin] RBAC: Authorizing `identity:create_grant(project_id=3002c2f1222949079ed30d3e444c6405, user_id=dcefd7a94ef2407b875e112b37e8116e, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:42.667550 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713494d7-cab8-49c0-aa11-9fbb57da1a3d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:42.680981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 897/1788] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:42 2022] PUT /identity/v3/projects/3002c2f1222949079ed30d3e444c6405/users/dcefd7a94ef2407b875e112b37e8116e/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:48:42.692608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd6b6ee-4f6a-4398-8e69-4d33676f6de9 service cinder] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:42.693779 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4cc51d-d4d9-45aa-9a3d-0a81251c833d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:42.694090 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4cc51d-d4d9-45aa-9a3d-0a81251c833d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:42.694355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4cc51d-d4d9-45aa-9a3d-0a81251c833d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:42.710815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd6b6ee-4f6a-4398-8e69-4d33676f6de9 service cinder] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:42.733527 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-eb4cc51d-d4d9-45aa-9a3d-0a81251c833d None None] MFA Rules not processed for user `dcefd7a94ef2407b875e112b37e8116e`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:42.734477 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-eb4cc51d-d4d9-45aa-9a3d-0a81251c833d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:42.825917 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2688aff-026d-43ba-a9a9-cb5bf26cb996 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:42.830691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2688aff-026d-43ba-a9a9-cb5bf26cb996 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:42.833623 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2688aff-026d-43ba-a9a9-cb5bf26cb996 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:42.834066 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2688aff-026d-43ba-a9a9-cb5bf26cb996 admin admin] PATH_INFO: `/v3/users/ecf17cdcc82d4c1e9376db42e1a20b1e` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:42.835835 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2688aff-026d-43ba-a9a9-cb5bf26cb996 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ecf17cdcc82d4c1e9376db42e1a20b1e)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:42.840861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2688aff-026d-43ba-a9a9-cb5bf26cb996 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:42.999925 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 898/1789] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:42 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:43.015683 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4cc51d-d4d9-45aa-9a3d-0a81251c833d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:43.169572 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c2688aff-026d-43ba-a9a9-cb5bf26cb996 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ecf17cdcc82d4c1e9376db42e1a20b1e'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:43.190230 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4cc51d-d4d9-45aa-9a3d-0a81251c833d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:43.201839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b353b6-3291-47d8-bda3-91c2bf055c32 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:43.203137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b353b6-3291-47d8-bda3-91c2bf055c32 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:43.204272 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b353b6-3291-47d8-bda3-91c2bf055c32 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:43.205027 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b353b6-3291-47d8-bda3-91c2bf055c32 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:43.231012 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b353b6-3291-47d8-bda3-91c2bf055c32 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:43.234859 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b353b6-3291-47d8-bda3-91c2bf055c32 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:43.280728 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 899/1790] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:43 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 266 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:43.283371 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2328f18-0d77-408c-aacd-052c1f6817a5 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:43.283716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2328f18-0d77-408c-aacd-052c1f6817a5 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:43.286110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2328f18-0d77-408c-aacd-052c1f6817a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:43.346251 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c2688aff-026d-43ba-a9a9-cb5bf26cb996 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ecf17cdcc82d4c1e9376db42e1a20b1e'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:43.387417 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c2688aff-026d-43ba-a9a9-cb5bf26cb996 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ecf17cdcc82d4c1e9376db42e1a20b1e'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:43.391781 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c2688aff-026d-43ba-a9a9-cb5bf26cb996 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ecf17cdcc82d4c1e9376db42e1a20b1e'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:43.421793 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c2688aff-026d-43ba-a9a9-cb5bf26cb996 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ecf17cdcc82d4c1e9376db42e1a20b1e'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:43.446813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 892/1791] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:48:42 2022] DELETE /identity/v3/users/ecf17cdcc82d4c1e9376db42e1a20b1e => generated 0 bytes in 755 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:43.457938 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb81d41-524a-439b-a241-f9e765bf9be6 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:43.458508 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb81d41-524a-439b-a241-f9e765bf9be6 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:43.461931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb81d41-524a-439b-a241-f9e765bf9be6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:43.489918 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f2328f18-0d77-408c-aacd-052c1f6817a5 None None] MFA Rules not processed for user `d40810f61ead4c3da414194b325da80a`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:43.491447 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f2328f18-0d77-408c-aacd-052c1f6817a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:43.536804 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-ddb81d41-524a-439b-a241-f9e765bf9be6 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:43.536804 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ddb81d41-524a-439b-a241-f9e765bf9be6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:43.711081 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 900/1792] 10.222.0.49 () {58 vars in 1009 bytes} [Fri Sep 16 03:48:43 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 429 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:43.734400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 893/1793] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:43.764503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2328f18-0d77-408c-aacd-052c1f6817a5 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:43.765981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f2328f18-0d77-408c-aacd-052c1f6817a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:43.870320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb81d41-524a-439b-a241-f9e765bf9be6 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:43.875267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ddb81d41-524a-439b-a241-f9e765bf9be6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:43.904576 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2328f18-0d77-408c-aacd-052c1f6817a5 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:43.909373 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a395d41-41a9-4649-bf99-46e1113ec754 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:43.910896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a395d41-41a9-4649-bf99-46e1113ec754 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:43.911311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a395d41-41a9-4649-bf99-46e1113ec754 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:43.911642 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a395d41-41a9-4649-bf99-46e1113ec754 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:43.931043 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a395d41-41a9-4649-bf99-46e1113ec754 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:43.935135 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a395d41-41a9-4649-bf99-46e1113ec754 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:43.974124 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 901/1794] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:48:44.022291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb81d41-524a-439b-a241-f9e765bf9be6 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:44.026412 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11b5b7e-7758-478e-976b-38d689f84a09 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:44.036726 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11b5b7e-7758-478e-976b-38d689f84a09 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:44.036726 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11b5b7e-7758-478e-976b-38d689f84a09 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:44.036726 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11b5b7e-7758-478e-976b-38d689f84a09 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:44.036726 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b11b5b7e-7758-478e-976b-38d689f84a09 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:44.067020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:DELETE=9,keystone:SELECT=89,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:44.111604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11b5b7e-7758-478e-976b-38d689f84a09 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:44.113147 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11b5b7e-7758-478e-976b-38d689f84a09 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:44.127071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 894/1795] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:43 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 259 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:48:44.237053 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a395d41-41a9-4649-bf99-46e1113ec754 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:44.239186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0a395d41-41a9-4649-bf99-46e1113ec754 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:44.428755 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a395d41-41a9-4649-bf99-46e1113ec754 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:44.442045 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffc2086-c5a3-4840-b161-516c581606c1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa30da9972e140d7905c522576681429', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:44.442684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffc2086-c5a3-4840-b161-516c581606c1 service swift] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:44.443013 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffc2086-c5a3-4840-b161-516c581606c1 service swift] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:44.443261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffc2086-c5a3-4840-b161-516c581606c1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:44.466862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffc2086-c5a3-4840-b161-516c581606c1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:44.468635 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffc2086-c5a3-4840-b161-516c581606c1 service swift] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:44.489294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 902/1796] 10.222.0.49 () {60 vars in 1384 bytes} [Fri Sep 16 03:48:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 253 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 03:48:44.605150 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c873afca-be4f-4c0b-95bc-752f02b65c34 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:44.605150 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c873afca-be4f-4c0b-95bc-752f02b65c34 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:44.605150 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c873afca-be4f-4c0b-95bc-752f02b65c34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:44.631632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-c873afca-be4f-4c0b-95bc-752f02b65c34 None None] MFA Rules not processed for user `f9275eeba471423e9fa2a31a6e08676d`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:44.639330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c873afca-be4f-4c0b-95bc-752f02b65c34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:44.642645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 895/1797] 10.222.0.52 () {58 vars in 1015 bytes} [Fri Sep 16 03:48:44 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:44.716907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffc2086-c5a3-4840-b161-516c581606c1 service swift] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:44.730038 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffc2086-c5a3-4840-b161-516c581606c1 service swift] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:44.774615 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3ada1f-14d4-4481-9ae5-134ef7d78bf1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:44.782782 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3ada1f-14d4-4481-9ae5-134ef7d78bf1 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:44.782782 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3ada1f-14d4-4481-9ae5-134ef7d78bf1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:44.782782 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3ada1f-14d4-4481-9ae5-134ef7d78bf1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:44.806440 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:44.816993 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3ada1f-14d4-4481-9ae5-134ef7d78bf1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:44.818461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3ada1f-14d4-4481-9ae5-134ef7d78bf1 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:44.855275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 903/1798] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:44 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:46.230235 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c873afca-be4f-4c0b-95bc-752f02b65c34 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:46.231732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c873afca-be4f-4c0b-95bc-752f02b65c34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:46.264755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c873afca-be4f-4c0b-95bc-752f02b65c34 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:46.270153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb9d8f4-1866-4d9d-acea-1ddf246ab464 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:46.271408 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb9d8f4-1866-4d9d-acea-1ddf246ab464 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:46.271857 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb9d8f4-1866-4d9d-acea-1ddf246ab464 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:46.272293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb9d8f4-1866-4d9d-acea-1ddf246ab464 admin admin] PATH_INFO: `/v3/projects/7bcc9b90fae04761a6382184e7381df4` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:46.273825 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb9d8f4-1866-4d9d-acea-1ddf246ab464 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7bcc9b90fae04761a6382184e7381df4)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:46.276032 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb9d8f4-1866-4d9d-acea-1ddf246ab464 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:46.593025 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-bbb9d8f4-1866-4d9d-acea-1ddf246ab464 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7bcc9b90fae04761a6382184e7381df4', 'request_id': 'req-bbb9d8f4-1866-4d9d-acea-1ddf246ab464'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:46.695449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-bbb9d8f4-1866-4d9d-acea-1ddf246ab464 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7bcc9b90fae04761a6382184e7381df4', 'request_id': 'req-bbb9d8f4-1866-4d9d-acea-1ddf246ab464'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:46.706572 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-bbb9d8f4-1866-4d9d-acea-1ddf246ab464 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7bcc9b90fae04761a6382184e7381df4', 'request_id': 'req-bbb9d8f4-1866-4d9d-acea-1ddf246ab464'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:46.722468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-bbb9d8f4-1866-4d9d-acea-1ddf246ab464 admin admin] The token cache is being invalidate because project 7bcc9b90fae04761a6382184e7381df4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:48:46.723194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-bbb9d8f4-1866-4d9d-acea-1ddf246ab464 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:46.724063 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-bbb9d8f4-1866-4d9d-acea-1ddf246ab464 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:46.735541 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3ada1f-14d4-4481-9ae5-134ef7d78bf1 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:46.759256 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 896/1799] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:48:46 2022] DELETE /identity/v3/projects/7bcc9b90fae04761a6382184e7381df4 => generated 0 bytes in 496 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:46.760004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3ada1f-14d4-4481-9ae5-134ef7d78bf1 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:46.902227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e9c8e7-4e09-476c-979f-fe153a114b35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:46.902227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e9c8e7-4e09-476c-979f-fe153a114b35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:46.902227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e9c8e7-4e09-476c-979f-fe153a114b35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:46.902227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e9c8e7-4e09-476c-979f-fe153a114b35 admin admin] PATH_INFO: `/v3/users/7843f2dc7efa453e9f65b8233d3e1419` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:46.902969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e9c8e7-4e09-476c-979f-fe153a114b35 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7843f2dc7efa453e9f65b8233d3e1419)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:46.920538 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e9c8e7-4e09-476c-979f-fe153a114b35 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:47.287637 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f3e9c8e7-4e09-476c-979f-fe153a114b35 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7843f2dc7efa453e9f65b8233d3e1419'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:47.287637 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f3e9c8e7-4e09-476c-979f-fe153a114b35 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7843f2dc7efa453e9f65b8233d3e1419'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:47.694094 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f3e9c8e7-4e09-476c-979f-fe153a114b35 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7843f2dc7efa453e9f65b8233d3e1419'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:47.695271 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f3e9c8e7-4e09-476c-979f-fe153a114b35 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7843f2dc7efa453e9f65b8233d3e1419'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:47.708709 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f3e9c8e7-4e09-476c-979f-fe153a114b35 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7843f2dc7efa453e9f65b8233d3e1419'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:47.728398 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 904/1800] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:48:46 2022] DELETE /identity/v3/users/7843f2dc7efa453e9f65b8233d3e1419 => generated 0 bytes in 995 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:48.402961 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09001f70-a0ff-4931-9579-12c167d075b9 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:48.405647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09001f70-a0ff-4931-9579-12c167d075b9 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:48.405647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09001f70-a0ff-4931-9579-12c167d075b9 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:48.407691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 897/1801] 10.222.0.49 () {54 vars in 921 bytes} [Fri Sep 16 03:48:48 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:48:48.487385 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7e252a-c356-400b-aa98-49b787d0d12b None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:48.488273 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7e252a-c356-400b-aa98-49b787d0d12b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:48.488717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7e252a-c356-400b-aa98-49b787d0d12b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:48.551676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-cb7e252a-c356-400b-aa98-49b787d0d12b None None] MFA Rules not processed for user `ded1bcd10e62499a9aaa0ece5aa55c80`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:48.552930 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cb7e252a-c356-400b-aa98-49b787d0d12b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:48.628383 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450b9f13-4cb7-43b6-8d53-1ec1e88f6510 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:48.628903 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450b9f13-4cb7-43b6-8d53-1ec1e88f6510 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:48.629338 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450b9f13-4cb7-43b6-8d53-1ec1e88f6510 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:48.630843 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 898/1802] 10.222.0.49 () {54 vars in 921 bytes} [Fri Sep 16 03:48:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:48:48.650628 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1825c651-a659-4fff-a002-3e62a7d2ed39 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:48.651195 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1825c651-a659-4fff-a002-3e62a7d2ed39 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:48.651644 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1825c651-a659-4fff-a002-3e62a7d2ed39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:48.704965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-1825c651-a659-4fff-a002-3e62a7d2ed39 None None] MFA Rules not processed for user `ded1bcd10e62499a9aaa0ece5aa55c80`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:48.706211 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1825c651-a659-4fff-a002-3e62a7d2ed39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:48.881074 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 905/1803] 10.222.0.49 () {58 vars in 1018 bytes} [Fri Sep 16 03:48:48 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 395 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:48.885227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7e252a-c356-400b-aa98-49b787d0d12b None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:48.896835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cb7e252a-c356-400b-aa98-49b787d0d12b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:48.948617 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 899/1804] 10.222.0.49 () {58 vars in 1018 bytes} [Fri Sep 16 03:48:48 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 299 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:48:49.245701 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7e252a-c356-400b-aa98-49b787d0d12b None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:49.267817 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56589847-d102-47f5-adbd-37f3d659c75e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:49.270360 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56589847-d102-47f5-adbd-37f3d659c75e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:49.270814 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56589847-d102-47f5-adbd-37f3d659c75e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:49.271322 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56589847-d102-47f5-adbd-37f3d659c75e admin admin] PATH_INFO: `/v3/users/e3e9ebc6fb2c4dac8ebdf3ad170bbfd3` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:49.273034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56589847-d102-47f5-adbd-37f3d659c75e admin admin] RBAC: Authorizing `identity:delete_user(user_id=e3e9ebc6fb2c4dac8ebdf3ad170bbfd3)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:49.276202 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56589847-d102-47f5-adbd-37f3d659c75e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:49.404007 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1825c651-a659-4fff-a002-3e62a7d2ed39 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:49.406145 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1825c651-a659-4fff-a002-3e62a7d2ed39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:49.422360 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1825c651-a659-4fff-a002-3e62a7d2ed39 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:49.429799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b93141-a81d-4996-9b11-488ea52bc6a6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:49.430978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b93141-a81d-4996-9b11-488ea52bc6a6 service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:49.431571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b93141-a81d-4996-9b11-488ea52bc6a6 service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:49.432005 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b93141-a81d-4996-9b11-488ea52bc6a6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:49.443865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b93141-a81d-4996-9b11-488ea52bc6a6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:49.446182 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b93141-a81d-4996-9b11-488ea52bc6a6 service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:49.460163 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 900/1805] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:49 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:48:49.704529 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:49.986117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-56589847-d102-47f5-adbd-37f3d659c75e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e3e9ebc6fb2c4dac8ebdf3ad170bbfd3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:50.020498 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-56589847-d102-47f5-adbd-37f3d659c75e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e3e9ebc6fb2c4dac8ebdf3ad170bbfd3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:50.094005 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b93141-a81d-4996-9b11-488ea52bc6a6 service nova] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:50.096396 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-f2b93141-a81d-4996-9b11-488ea52bc6a6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:50.096760 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-56589847-d102-47f5-adbd-37f3d659c75e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e3e9ebc6fb2c4dac8ebdf3ad170bbfd3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:50.097458 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-56589847-d102-47f5-adbd-37f3d659c75e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e3e9ebc6fb2c4dac8ebdf3ad170bbfd3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:50.110600 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-56589847-d102-47f5-adbd-37f3d659c75e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e3e9ebc6fb2c4dac8ebdf3ad170bbfd3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:50.124062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 906/1806] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:48:48 2022] DELETE /identity/v3/users/e3e9ebc6fb2c4dac8ebdf3ad170bbfd3 => generated 0 bytes in 1241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:50.125448 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56589847-d102-47f5-adbd-37f3d659c75e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:50.128799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-56589847-d102-47f5-adbd-37f3d659c75e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:50.129349 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b93141-a81d-4996-9b11-488ea52bc6a6 service nova] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:50.276023 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4c8c87-e491-4c88-8469-38f5c2093719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:50.276023 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4c8c87-e491-4c88-8469-38f5c2093719 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:50.276023 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4c8c87-e491-4c88-8469-38f5c2093719 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:50.276569 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4c8c87-e491-4c88-8469-38f5c2093719 admin admin] PATH_INFO: `/v3/projects/768e634e5f60449eb49465798f8c167a` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:50.278713 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4c8c87-e491-4c88-8469-38f5c2093719 admin admin] RBAC: Authorizing `identity:delete_project(project_id=768e634e5f60449eb49465798f8c167a)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:50.282574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4c8c87-e491-4c88-8469-38f5c2093719 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:50.368721 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56589847-d102-47f5-adbd-37f3d659c75e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:50.370208 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373b4ec9-3b6b-4f41-8a92-7ca84efa4075 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:50.371225 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373b4ec9-3b6b-4f41-8a92-7ca84efa4075 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:50.371699 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373b4ec9-3b6b-4f41-8a92-7ca84efa4075 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:50.372499 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373b4ec9-3b6b-4f41-8a92-7ca84efa4075 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:50.390624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-373b4ec9-3b6b-4f41-8a92-7ca84efa4075 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:50.474998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373b4ec9-3b6b-4f41-8a92-7ca84efa4075 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:50.474998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373b4ec9-3b6b-4f41-8a92-7ca84efa4075 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:50.502368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 907/1807] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:50 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:50.503631 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-2d4c8c87-e491-4c88-8469-38f5c2093719 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '768e634e5f60449eb49465798f8c167a', 'request_id': 'req-2d4c8c87-e491-4c88-8469-38f5c2093719'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:50.527588 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9540c946-a560-476e-9aee-be1055b95a5f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:50.527881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9540c946-a560-476e-9aee-be1055b95a5f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:50.534601 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9540c946-a560-476e-9aee-be1055b95a5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:50.542674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:50.578804 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-2d4c8c87-e491-4c88-8469-38f5c2093719 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '768e634e5f60449eb49465798f8c167a', 'request_id': 'req-2d4c8c87-e491-4c88-8469-38f5c2093719'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:50.588667 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-2d4c8c87-e491-4c88-8469-38f5c2093719 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '768e634e5f60449eb49465798f8c167a', 'request_id': 'req-2d4c8c87-e491-4c88-8469-38f5c2093719'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:50.607947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9540c946-a560-476e-9aee-be1055b95a5f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:50.609751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9540c946-a560-476e-9aee-be1055b95a5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:50.627691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-2d4c8c87-e491-4c88-8469-38f5c2093719 admin admin] The token cache is being invalidate because project 768e634e5f60449eb49465798f8c167a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:48:50.631488 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-2d4c8c87-e491-4c88-8469-38f5c2093719 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:50.633676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-2d4c8c87-e491-4c88-8469-38f5c2093719 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:50.642090 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 901/1808] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:48:50 2022] DELETE /identity/v3/projects/768e634e5f60449eb49465798f8c167a => generated 0 bytes in 545 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:48:50.644209 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6363720b-f619-4f26-896e-0003ba2d3587 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:50.645187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6363720b-f619-4f26-896e-0003ba2d3587 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:50.646252 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6363720b-f619-4f26-896e-0003ba2d3587 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:50.704337 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-6363720b-f619-4f26-896e-0003ba2d3587 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:50.705820 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-6363720b-f619-4f26-896e-0003ba2d3587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:50.737629 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 908/1809] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:50.748127 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9540c946-a560-476e-9aee-be1055b95a5f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:50.750247 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9540c946-a560-476e-9aee-be1055b95a5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:50.811684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 902/1810] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:50.821298 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf202adc-1b39-497f-9e73-884809597a2c None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:50.821608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf202adc-1b39-497f-9e73-884809597a2c None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:50.821934 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf202adc-1b39-497f-9e73-884809597a2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:50.854724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9540c946-a560-476e-9aee-be1055b95a5f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:50.859476 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d679e67-4fbf-49d9-b4c9-64e7d4c32f8e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:50.860333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d679e67-4fbf-49d9-b4c9-64e7d4c32f8e service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:50.860806 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d679e67-4fbf-49d9-b4c9-64e7d4c32f8e service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:50.861180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d679e67-4fbf-49d9-b4c9-64e7d4c32f8e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:50.884514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-bf202adc-1b39-497f-9e73-884809597a2c None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:50.885944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d679e67-4fbf-49d9-b4c9-64e7d4c32f8e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:50.888404 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-bf202adc-1b39-497f-9e73-884809597a2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:50.891217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d679e67-4fbf-49d9-b4c9-64e7d4c32f8e service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:50.918308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 903/1811] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:50.918308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf202adc-1b39-497f-9e73-884809597a2c None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:50.929825 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 909/1812] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:48:50.977001 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf202adc-1b39-497f-9e73-884809597a2c None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:50.992871 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d243f9c-de0c-4887-9bd2-fc8c85840f66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:50.992871 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d243f9c-de0c-4887-9bd2-fc8c85840f66 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:50.993341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d243f9c-de0c-4887-9bd2-fc8c85840f66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:50.999262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d243f9c-de0c-4887-9bd2-fc8c85840f66 admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:51.004141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d243f9c-de0c-4887-9bd2-fc8c85840f66 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:48:51.005908 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d243f9c-de0c-4887-9bd2-fc8c85840f66 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:51.014416 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d243f9c-de0c-4887-9bd2-fc8c85840f66 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:51.042560 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 904/1813] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:48:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:48:51.048903 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d679e67-4fbf-49d9-b4c9-64e7d4c32f8e service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:51.063410 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d679e67-4fbf-49d9-b4c9-64e7d4c32f8e service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:51.071644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a19c19-e5f8-4336-b970-e4d8b94f41ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:51.072890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a19c19-e5f8-4336-b970-e4d8b94f41ba admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:51.073817 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a19c19-e5f8-4336-b970-e4d8b94f41ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:51.074110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a19c19-e5f8-4336-b970-e4d8b94f41ba admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:51.074803 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a19c19-e5f8-4336-b970-e4d8b94f41ba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:51.078169 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a19c19-e5f8-4336-b970-e4d8b94f41ba admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:51.353270 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d243f9c-de0c-4887-9bd2-fc8c85840f66 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:51.354585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 910/1814] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:51 2022] POST /identity/v3/projects => generated 383 bytes in 306 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:51.380861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d243f9c-de0c-4887-9bd2-fc8c85840f66 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:51.540725 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69410aa0-e755-4e32-901b-2780393182b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:51.540725 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69410aa0-e755-4e32-901b-2780393182b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:51.540725 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69410aa0-e755-4e32-901b-2780393182b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:51.541992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69410aa0-e755-4e32-901b-2780393182b0 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:51.542934 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69410aa0-e755-4e32-901b-2780393182b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:51.548094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69410aa0-e755-4e32-901b-2780393182b0 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:51.759638 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 905/1815] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:51 2022] POST /identity/v3/users => generated 292 bytes in 407 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:51.760440 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a19c19-e5f8-4336-b970-e4d8b94f41ba admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:51.787618 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a19c19-e5f8-4336-b970-e4d8b94f41ba admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:51.816184 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e61d7c4-8361-467e-922b-866b058fbe4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:51.816819 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e61d7c4-8361-467e-922b-866b058fbe4e admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:51.817171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e61d7c4-8361-467e-922b-866b058fbe4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:51.817550 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e61d7c4-8361-467e-922b-866b058fbe4e admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:51.818137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e61d7c4-8361-467e-922b-866b058fbe4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:51.819993 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e61d7c4-8361-467e-922b-866b058fbe4e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:51.858855 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 911/1816] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:51 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:48:51.860806 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69410aa0-e755-4e32-901b-2780393182b0 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:51.876686 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69410aa0-e755-4e32-901b-2780393182b0 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:51.893604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4419bb1f-8096-4227-8f74-ded160fe7bb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:51.894161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4419bb1f-8096-4227-8f74-ded160fe7bb0 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:51.894515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4419bb1f-8096-4227-8f74-ded160fe7bb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:51.895021 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4419bb1f-8096-4227-8f74-ded160fe7bb0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:51.899094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4419bb1f-8096-4227-8f74-ded160fe7bb0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:51.899094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4419bb1f-8096-4227-8f74-ded160fe7bb0 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:51.940806 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 906/1817] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:51 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:51.946004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e61d7c4-8361-467e-922b-866b058fbe4e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:51.959954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e61d7c4-8361-467e-922b-866b058fbe4e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:51.979050 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31430a3c-ba61-47b7-bd11-885267b2c50f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:51.979841 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31430a3c-ba61-47b7-bd11-885267b2c50f admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:51.980161 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31430a3c-ba61-47b7-bd11-885267b2c50f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:51.980393 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31430a3c-ba61-47b7-bd11-885267b2c50f admin admin] PATH_INFO: `/v3/projects/fdcbab241b3b464dbb487a455fbd29f8/users/4dce564b8a1447d5970e4887507d825d/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:51.999578 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:52.048282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31430a3c-ba61-47b7-bd11-885267b2c50f admin admin] RBAC: Authorizing `identity:create_grant(project_id=fdcbab241b3b464dbb487a455fbd29f8, user_id=4dce564b8a1447d5970e4887507d825d, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:52.051538 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31430a3c-ba61-47b7-bd11-885267b2c50f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:52.395674 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 912/1818] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:51 2022] PUT /identity/v3/projects/fdcbab241b3b464dbb487a455fbd29f8/users/4dce564b8a1447d5970e4887507d825d/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 427 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:48:52.399928 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdd1d80-aae6-48b5-8075-806d41a064ab None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:52.400470 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdd1d80-aae6-48b5-8075-806d41a064ab None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:52.400901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdd1d80-aae6-48b5-8075-806d41a064ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:52.452650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-acdd1d80-aae6-48b5-8075-806d41a064ab None None] MFA Rules not processed for user `4dce564b8a1447d5970e4887507d825d`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:52.454277 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-acdd1d80-aae6-48b5-8075-806d41a064ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:52.631972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 907/1819] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:52 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:52.636869 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31430a3c-ba61-47b7-bd11-885267b2c50f admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:52.831180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31430a3c-ba61-47b7-bd11-885267b2c50f admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:52.834030 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903c2e47-1579-4e8d-903c-370fe85d0511 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:52.834943 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903c2e47-1579-4e8d-903c-370fe85d0511 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:52.835421 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903c2e47-1579-4e8d-903c-370fe85d0511 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:52.835929 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903c2e47-1579-4e8d-903c-370fe85d0511 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:52.836811 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903c2e47-1579-4e8d-903c-370fe85d0511 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:52.839722 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903c2e47-1579-4e8d-903c-370fe85d0511 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:53.000578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 913/1820] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:52 2022] POST /identity/v3/projects => generated 381 bytes in 364 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:53.004745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdd1d80-aae6-48b5-8075-806d41a064ab None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:53.031026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdd1d80-aae6-48b5-8075-806d41a064ab None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:53.038229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903c2e47-1579-4e8d-903c-370fe85d0511 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:53.040843 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-903c2e47-1579-4e8d-903c-370fe85d0511 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:53.286186 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9317b559-6744-4bd7-83c9-ba236b2d108d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:53.286994 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9317b559-6744-4bd7-83c9-ba236b2d108d admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:53.287345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9317b559-6744-4bd7-83c9-ba236b2d108d admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:53.288107 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9317b559-6744-4bd7-83c9-ba236b2d108d admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:53.288803 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9317b559-6744-4bd7-83c9-ba236b2d108d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:53.291835 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9317b559-6744-4bd7-83c9-ba236b2d108d admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:53.309279 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903c2e47-1579-4e8d-903c-370fe85d0511 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:53.328801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f56872b-c9a3-44fa-bca9-1f22654fc0d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:53.329874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f56872b-c9a3-44fa-bca9-1f22654fc0d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:53.332183 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f56872b-c9a3-44fa-bca9-1f22654fc0d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:53.332673 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f56872b-c9a3-44fa-bca9-1f22654fc0d9 admin admin] PATH_INFO: `/v3/users/2b35fa35fcf140ec8299992b60dca043` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:53.340451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f56872b-c9a3-44fa-bca9-1f22654fc0d9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2b35fa35fcf140ec8299992b60dca043)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:53.342826 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f56872b-c9a3-44fa-bca9-1f22654fc0d9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:53.476500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 908/1821] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:53 2022] POST /identity/v3/users => generated 297 bytes in 466 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:53.476500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9317b559-6744-4bd7-83c9-ba236b2d108d admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:53.476500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-9317b559-6744-4bd7-83c9-ba236b2d108d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:53.503064 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9317b559-6744-4bd7-83c9-ba236b2d108d admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:53.505682 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:53.508621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd0f018-a97a-4c5f-b725-1f334b4af373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:53.510057 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd0f018-a97a-4c5f-b725-1f334b4af373 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:53.510057 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd0f018-a97a-4c5f-b725-1f334b4af373 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:53.510057 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd0f018-a97a-4c5f-b725-1f334b4af373 admin admin] PATH_INFO: `/v3/projects/04f53e174db94e41bac3e45226a715e5` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:53.521446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd0f018-a97a-4c5f-b725-1f334b4af373 admin admin] RBAC: Authorizing `identity:delete_project(project_id=04f53e174db94e41bac3e45226a715e5)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:53.523186 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd0f018-a97a-4c5f-b725-1f334b4af373 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:53.624318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7f56872b-c9a3-44fa-bca9-1f22654fc0d9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2b35fa35fcf140ec8299992b60dca043'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:53.666513 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7f56872b-c9a3-44fa-bca9-1f22654fc0d9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2b35fa35fcf140ec8299992b60dca043'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:53.785939 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-3dd0f018-a97a-4c5f-b725-1f334b4af373 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '04f53e174db94e41bac3e45226a715e5', 'request_id': 'req-3dd0f018-a97a-4c5f-b725-1f334b4af373'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:53.793792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7f56872b-c9a3-44fa-bca9-1f22654fc0d9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2b35fa35fcf140ec8299992b60dca043'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:53.794539 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7f56872b-c9a3-44fa-bca9-1f22654fc0d9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2b35fa35fcf140ec8299992b60dca043'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:53.811054 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7f56872b-c9a3-44fa-bca9-1f22654fc0d9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2b35fa35fcf140ec8299992b60dca043'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:53.836225 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 914/1822] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:48:53 2022] DELETE /identity/v3/users/2b35fa35fcf140ec8299992b60dca043 => generated 0 bytes in 799 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:48:53.844578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f56872b-c9a3-44fa-bca9-1f22654fc0d9 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:53.863509 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7f56872b-c9a3-44fa-bca9-1f22654fc0d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:53.942367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-3dd0f018-a97a-4c5f-b725-1f334b4af373 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '04f53e174db94e41bac3e45226a715e5', 'request_id': 'req-3dd0f018-a97a-4c5f-b725-1f334b4af373'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:53.949908 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-3dd0f018-a97a-4c5f-b725-1f334b4af373 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '04f53e174db94e41bac3e45226a715e5', 'request_id': 'req-3dd0f018-a97a-4c5f-b725-1f334b4af373'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:53.991088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-3dd0f018-a97a-4c5f-b725-1f334b4af373 admin admin] The token cache is being invalidate because project 04f53e174db94e41bac3e45226a715e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:48:53.991497 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-3dd0f018-a97a-4c5f-b725-1f334b4af373 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:53.996194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-3dd0f018-a97a-4c5f-b725-1f334b4af373 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:54.004672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 909/1823] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:48:53 2022] DELETE /identity/v3/projects/04f53e174db94e41bac3e45226a715e5 => generated 0 bytes in 530 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:54.004672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed6ea05-742e-4562-ac14-52be3efb9fd5 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:54.004672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed6ea05-742e-4562-ac14-52be3efb9fd5 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:54.004672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed6ea05-742e-4562-ac14-52be3efb9fd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:54.016320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f56872b-c9a3-44fa-bca9-1f22654fc0d9 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:54.023410 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a931eea-ef5d-46a9-a4d7-cdf02459414a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:54.024089 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a931eea-ef5d-46a9-a4d7-cdf02459414a admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:54.024378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a931eea-ef5d-46a9-a4d7-cdf02459414a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:54.056624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a931eea-ef5d-46a9-a4d7-cdf02459414a admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:54.058459 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a931eea-ef5d-46a9-a4d7-cdf02459414a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:54.064474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a931eea-ef5d-46a9-a4d7-cdf02459414a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:54.080063 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-9ed6ea05-742e-4562-ac14-52be3efb9fd5 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:54.081706 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-9ed6ea05-742e-4562-ac14-52be3efb9fd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:54.116226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 915/1824] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:53 2022] GET /identity/v3/roles => generated 1420 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:54.121056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a931eea-ef5d-46a9-a4d7-cdf02459414a admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:54.138410 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6a931eea-ef5d-46a9-a4d7-cdf02459414a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:54.143347 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 910/1825] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:54.144621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed6ea05-742e-4562-ac14-52be3efb9fd5 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:54.151569 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-9ed6ea05-742e-4562-ac14-52be3efb9fd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:54.169054 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a931eea-ef5d-46a9-a4d7-cdf02459414a admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:54.188156 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed6ea05-742e-4562-ac14-52be3efb9fd5 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:54.199663 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e99356b-32d9-4bcc-bff5-f3d8a111dfd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:54.200682 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e99356b-32d9-4bcc-bff5-f3d8a111dfd1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:54.202035 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e99356b-32d9-4bcc-bff5-f3d8a111dfd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:54.202477 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e99356b-32d9-4bcc-bff5-f3d8a111dfd1 admin admin] PATH_INFO: `/v3/users/432b4d98e9b54d0ebad16a3b3d854b72` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:54.205648 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e99356b-32d9-4bcc-bff5-f3d8a111dfd1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=432b4d98e9b54d0ebad16a3b3d854b72)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:54.224997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e99356b-32d9-4bcc-bff5-f3d8a111dfd1 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:54.227835 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eed3909-e776-4bb9-a8b3-fd32b624ee60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:54.231981 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eed3909-e776-4bb9-a8b3-fd32b624ee60 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:54.231981 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eed3909-e776-4bb9-a8b3-fd32b624ee60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:54.231981 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eed3909-e776-4bb9-a8b3-fd32b624ee60 admin admin] PATH_INFO: `/v3/projects/e8f0e60e237c480b8bdd1374d0f8ec53/users/57e9eb7c3277426294bcd1ccdb813aba/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:54.272630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eed3909-e776-4bb9-a8b3-fd32b624ee60 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8f0e60e237c480b8bdd1374d0f8ec53, user_id=57e9eb7c3277426294bcd1ccdb813aba, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:54.275634 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eed3909-e776-4bb9-a8b3-fd32b624ee60 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:54.482677 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 911/1826] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:54 2022] PUT /identity/v3/projects/e8f0e60e237c480b8bdd1374d0f8ec53/users/57e9eb7c3277426294bcd1ccdb813aba/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:54.484449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eed3909-e776-4bb9-a8b3-fd32b624ee60 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:54.488811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3eed3909-e776-4bb9-a8b3-fd32b624ee60 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:54.579595 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8e99356b-32d9-4bcc-bff5-f3d8a111dfd1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '432b4d98e9b54d0ebad16a3b3d854b72'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:54.614023 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eed3909-e776-4bb9-a8b3-fd32b624ee60 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:54.619365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bd681f-c2ad-49dc-875a-c6d816492510 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:54.620222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bd681f-c2ad-49dc-875a-c6d816492510 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:54.620650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bd681f-c2ad-49dc-875a-c6d816492510 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:54.621095 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bd681f-c2ad-49dc-875a-c6d816492510 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:54.622637 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8e99356b-32d9-4bcc-bff5-f3d8a111dfd1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '432b4d98e9b54d0ebad16a3b3d854b72'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:54.723369 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2bd681f-c2ad-49dc-875a-c6d816492510 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:54.724848 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2bd681f-c2ad-49dc-875a-c6d816492510 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:54.727778 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8e99356b-32d9-4bcc-bff5-f3d8a111dfd1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '432b4d98e9b54d0ebad16a3b3d854b72'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:54.728575 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8e99356b-32d9-4bcc-bff5-f3d8a111dfd1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '432b4d98e9b54d0ebad16a3b3d854b72'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:54.750357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8e99356b-32d9-4bcc-bff5-f3d8a111dfd1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '432b4d98e9b54d0ebad16a3b3d854b72'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:54.781691 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 916/1827] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:48:54 2022] DELETE /identity/v3/users/432b4d98e9b54d0ebad16a3b3d854b72 => generated 0 bytes in 649 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:54.782787 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e99356b-32d9-4bcc-bff5-f3d8a111dfd1 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:54.790150 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 912/1828] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:54.792477 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8e99356b-32d9-4bcc-bff5-f3d8a111dfd1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:54.956426 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e99356b-32d9-4bcc-bff5-f3d8a111dfd1 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:54.964041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76500609-ca2d-4e06-ab9f-8bc547d7f34a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:54.964780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76500609-ca2d-4e06-ab9f-8bc547d7f34a admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:54.965320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76500609-ca2d-4e06-ab9f-8bc547d7f34a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:54.965575 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76500609-ca2d-4e06-ab9f-8bc547d7f34a admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:54.966674 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76500609-ca2d-4e06-ab9f-8bc547d7f34a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:54.971071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76500609-ca2d-4e06-ab9f-8bc547d7f34a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:54.994729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 917/1829] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:54 2022] GET /identity/v3/roles => generated 1420 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:54.999291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bd681f-c2ad-49dc-875a-c6d816492510 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:55.015447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bd681f-c2ad-49dc-875a-c6d816492510 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:55.027652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eae866b-0e33-4f23-87a0-0ebb83d266e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:55.027652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eae866b-0e33-4f23-87a0-0ebb83d266e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:55.027652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eae866b-0e33-4f23-87a0-0ebb83d266e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:55.027652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eae866b-0e33-4f23-87a0-0ebb83d266e3 admin admin] PATH_INFO: `/v3/projects/e8f0e60e237c480b8bdd1374d0f8ec53/users/57e9eb7c3277426294bcd1ccdb813aba/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:55.028030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eae866b-0e33-4f23-87a0-0ebb83d266e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8f0e60e237c480b8bdd1374d0f8ec53, user_id=57e9eb7c3277426294bcd1ccdb813aba, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:55.031681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eae866b-0e33-4f23-87a0-0ebb83d266e3 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:55.052698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 913/1830] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:54 2022] PUT /identity/v3/projects/e8f0e60e237c480b8bdd1374d0f8ec53/users/57e9eb7c3277426294bcd1ccdb813aba/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:48:55.064300 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59df8a1c-11b1-4586-aaea-daa9581fbc7e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:55.064975 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59df8a1c-11b1-4586-aaea-daa9581fbc7e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:55.065487 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59df8a1c-11b1-4586-aaea-daa9581fbc7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:55.117004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-59df8a1c-11b1-4586-aaea-daa9581fbc7e None None] MFA Rules not processed for user `57e9eb7c3277426294bcd1ccdb813aba`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:55.118474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-59df8a1c-11b1-4586-aaea-daa9581fbc7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:55.294154 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 918/1831] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:55 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:55.339895 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eae866b-0e33-4f23-87a0-0ebb83d266e3 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:55.340116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-2eae866b-0e33-4f23-87a0-0ebb83d266e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:55.444199 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eae866b-0e33-4f23-87a0-0ebb83d266e3 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:55.450442 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9f5101-503c-4a74-b1fb-b34bfe61500a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:55.452231 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9f5101-503c-4a74-b1fb-b34bfe61500a service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:55.453214 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9f5101-503c-4a74-b1fb-b34bfe61500a service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:55.453793 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9f5101-503c-4a74-b1fb-b34bfe61500a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:55.486721 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9f5101-503c-4a74-b1fb-b34bfe61500a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:55.488920 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9f5101-503c-4a74-b1fb-b34bfe61500a service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:55.541509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 914/1832] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:55 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:55.678055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59df8a1c-11b1-4586-aaea-daa9581fbc7e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:55.823772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59df8a1c-11b1-4586-aaea-daa9581fbc7e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:55.830652 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f77ec6-667d-492f-86f4-7f37eec04dc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:55.833748 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f77ec6-667d-492f-86f4-7f37eec04dc0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:55.833748 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f77ec6-667d-492f-86f4-7f37eec04dc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:55.833748 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f77ec6-667d-492f-86f4-7f37eec04dc0 admin admin] PATH_INFO: `/v3/users/4dce564b8a1447d5970e4887507d825d` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:55.833748 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f77ec6-667d-492f-86f4-7f37eec04dc0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4dce564b8a1447d5970e4887507d825d)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:55.845645 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f77ec6-667d-492f-86f4-7f37eec04dc0 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:56.200892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f7f77ec6-667d-492f-86f4-7f37eec04dc0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4dce564b8a1447d5970e4887507d825d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:56.228369 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f7f77ec6-667d-492f-86f4-7f37eec04dc0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4dce564b8a1447d5970e4887507d825d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:56.417238 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f7f77ec6-667d-492f-86f4-7f37eec04dc0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4dce564b8a1447d5970e4887507d825d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:56.418311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f7f77ec6-667d-492f-86f4-7f37eec04dc0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4dce564b8a1447d5970e4887507d825d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:56.435997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f7f77ec6-667d-492f-86f4-7f37eec04dc0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4dce564b8a1447d5970e4887507d825d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:56.461386 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 919/1833] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:48:55 2022] DELETE /identity/v3/users/4dce564b8a1447d5970e4887507d825d => generated 0 bytes in 784 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:56.465762 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2767b419-0f20-4db4-92fa-d97d034a31a1 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:56.466273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2767b419-0f20-4db4-92fa-d97d034a31a1 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:56.466273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2767b419-0f20-4db4-92fa-d97d034a31a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:56.524142 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-2767b419-0f20-4db4-92fa-d97d034a31a1 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:56.525400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-2767b419-0f20-4db4-92fa-d97d034a31a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:56.615501 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 915/1834] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:56.647920 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f77ec6-667d-492f-86f4-7f37eec04dc0 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:56.651213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f7f77ec6-667d-492f-86f4-7f37eec04dc0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:56.871871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f77ec6-667d-492f-86f4-7f37eec04dc0 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:56.882706 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187875cb-679a-4267-8fd9-7ce7390c18c2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:56.891612 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187875cb-679a-4267-8fd9-7ce7390c18c2 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:56.891612 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187875cb-679a-4267-8fd9-7ce7390c18c2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:56.891612 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187875cb-679a-4267-8fd9-7ce7390c18c2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:56.901281 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187875cb-679a-4267-8fd9-7ce7390c18c2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:56.903834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187875cb-679a-4267-8fd9-7ce7390c18c2 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:56.928288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 920/1835] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 281 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:48:57.044447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2767b419-0f20-4db4-92fa-d97d034a31a1 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:57.047548 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-2767b419-0f20-4db4-92fa-d97d034a31a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:57.067434 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2767b419-0f20-4db4-92fa-d97d034a31a1 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:57.079088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f019cf8-565b-4cec-af90-d11c58d70327 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:57.079759 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f019cf8-565b-4cec-af90-d11c58d70327 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:57.080038 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f019cf8-565b-4cec-af90-d11c58d70327 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:57.080420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f019cf8-565b-4cec-af90-d11c58d70327 admin admin] PATH_INFO: `/v3/projects/b3e694df24814a758811e7b2f482c808` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:57.083097 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f019cf8-565b-4cec-af90-d11c58d70327 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b3e694df24814a758811e7b2f482c808)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:57.087148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f019cf8-565b-4cec-af90-d11c58d70327 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:57.230406 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187875cb-679a-4267-8fd9-7ce7390c18c2 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:57.232097 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-187875cb-679a-4267-8fd9-7ce7390c18c2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:57.251280 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-8f019cf8-565b-4cec-af90-d11c58d70327 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b3e694df24814a758811e7b2f482c808', 'request_id': 'req-8f019cf8-565b-4cec-af90-d11c58d70327'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:57.253423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187875cb-679a-4267-8fd9-7ce7390c18c2 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:57.362365 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59dc925-e8f4-4c99-94a4-9f237b6a23bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:57.368027 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59dc925-e8f4-4c99-94a4-9f237b6a23bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:57.368432 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59dc925-e8f4-4c99-94a4-9f237b6a23bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:57.368639 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59dc925-e8f4-4c99-94a4-9f237b6a23bc admin admin] PATH_INFO: `/v3/projects/7f7b6c5700b3450b8cc88efe291af9c7` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:57.373791 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59dc925-e8f4-4c99-94a4-9f237b6a23bc admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f7b6c5700b3450b8cc88efe291af9c7)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:57.375866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59dc925-e8f4-4c99-94a4-9f237b6a23bc admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:57.399293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-8f019cf8-565b-4cec-af90-d11c58d70327 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b3e694df24814a758811e7b2f482c808', 'request_id': 'req-8f019cf8-565b-4cec-af90-d11c58d70327'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:57.409697 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-8f019cf8-565b-4cec-af90-d11c58d70327 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b3e694df24814a758811e7b2f482c808', 'request_id': 'req-8f019cf8-565b-4cec-af90-d11c58d70327'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:57.443095 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-8f019cf8-565b-4cec-af90-d11c58d70327 admin admin] The token cache is being invalidate because project b3e694df24814a758811e7b2f482c808 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:48:57.443095 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-8f019cf8-565b-4cec-af90-d11c58d70327 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:57.443565 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-8f019cf8-565b-4cec-af90-d11c58d70327 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:57.445324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 916/1836] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:48:57 2022] DELETE /identity/v3/projects/b3e694df24814a758811e7b2f482c808 => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:57.453567 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795505c7-0873-42af-ac9f-9e10854b300a None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:57.454252 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795505c7-0873-42af-ac9f-9e10854b300a None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:57.454785 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795505c7-0873-42af-ac9f-9e10854b300a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:57.478852 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-795505c7-0873-42af-ac9f-9e10854b300a None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:57.481093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-795505c7-0873-42af-ac9f-9e10854b300a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:57.494532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 917/1837] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:57.506011 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10fd511-e167-4c72-b349-50ad319a1ede None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:57.506456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10fd511-e167-4c72-b349-50ad319a1ede None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:57.506930 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10fd511-e167-4c72-b349-50ad319a1ede None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:57.531270 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-b10fd511-e167-4c72-b349-50ad319a1ede None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:57.532651 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b10fd511-e167-4c72-b349-50ad319a1ede None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:57.545720 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 918/1838] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:48:57.550245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10fd511-e167-4c72-b349-50ad319a1ede None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:57.563992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f59dc925-e8f4-4c99-94a4-9f237b6a23bc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f7b6c5700b3450b8cc88efe291af9c7', 'request_id': 'req-f59dc925-e8f4-4c99-94a4-9f237b6a23bc'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:57.570964 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10fd511-e167-4c72-b349-50ad319a1ede None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:57.589735 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f59dc925-e8f4-4c99-94a4-9f237b6a23bc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f7b6c5700b3450b8cc88efe291af9c7', 'request_id': 'req-f59dc925-e8f4-4c99-94a4-9f237b6a23bc'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:57.650641 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd597f5-f21f-4f24-ac57-0778132b844b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:57.651563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd597f5-f21f-4f24-ac57-0778132b844b admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:57.652115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd597f5-f21f-4f24-ac57-0778132b844b admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:57.652523 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd597f5-f21f-4f24-ac57-0778132b844b admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:57.653333 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd597f5-f21f-4f24-ac57-0778132b844b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:48:57.653780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd597f5-f21f-4f24-ac57-0778132b844b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:57.655835 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd597f5-f21f-4f24-ac57-0778132b844b admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:57.675992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 919/1839] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:48:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:48:57.680630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd597f5-f21f-4f24-ac57-0778132b844b admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:57.690949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd597f5-f21f-4f24-ac57-0778132b844b admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:57.704442 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a6341b-24e0-43d8-8e18-dfaa9d368ff8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:57.705100 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a6341b-24e0-43d8-8e18-dfaa9d368ff8 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:57.705376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a6341b-24e0-43d8-8e18-dfaa9d368ff8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:57.705651 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a6341b-24e0-43d8-8e18-dfaa9d368ff8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:57.706317 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a6341b-24e0-43d8-8e18-dfaa9d368ff8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:57.708332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a6341b-24e0-43d8-8e18-dfaa9d368ff8 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:57.765120 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f59dc925-e8f4-4c99-94a4-9f237b6a23bc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f7b6c5700b3450b8cc88efe291af9c7', 'request_id': 'req-f59dc925-e8f4-4c99-94a4-9f237b6a23bc'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:57.792570 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f59dc925-e8f4-4c99-94a4-9f237b6a23bc admin admin] The token cache is being invalidate because project 7f7b6c5700b3450b8cc88efe291af9c7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:48:57.792998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f59dc925-e8f4-4c99-94a4-9f237b6a23bc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:57.796386 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f59dc925-e8f4-4c99-94a4-9f237b6a23bc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:57.800730 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 921/1840] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:48:57 2022] DELETE /identity/v3/projects/7f7b6c5700b3450b8cc88efe291af9c7 => generated 0 bytes in 569 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:57.804367 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59dc925-e8f4-4c99-94a4-9f237b6a23bc admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:57.806777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f59dc925-e8f4-4c99-94a4-9f237b6a23bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:57.829848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59dc925-e8f4-4c99-94a4-9f237b6a23bc admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:57.838367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 920/1841] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:57 2022] POST /identity/v3/projects => generated 367 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:48:57.839826 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24396b0e-e194-4359-875f-b2dab908796a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:57.840761 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24396b0e-e194-4359-875f-b2dab908796a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:57.841229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24396b0e-e194-4359-875f-b2dab908796a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:57.842109 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24396b0e-e194-4359-875f-b2dab908796a admin admin] PATH_INFO: `/v3/users/dcefd7a94ef2407b875e112b37e8116e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:57.842273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a6341b-24e0-43d8-8e18-dfaa9d368ff8 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:57.843961 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24396b0e-e194-4359-875f-b2dab908796a admin admin] RBAC: Authorizing `identity:delete_user(user_id=dcefd7a94ef2407b875e112b37e8116e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:57.847173 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24396b0e-e194-4359-875f-b2dab908796a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:57.848708 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c4a6341b-24e0-43d8-8e18-dfaa9d368ff8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:57.925801 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a6341b-24e0-43d8-8e18-dfaa9d368ff8 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:57.931268 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fc3fdb-64e6-460a-80f5-a477cdcd0a58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:57.932524 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fc3fdb-64e6-460a-80f5-a477cdcd0a58 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:57.932865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fc3fdb-64e6-460a-80f5-a477cdcd0a58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:57.933171 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fc3fdb-64e6-460a-80f5-a477cdcd0a58 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:57.933958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74fc3fdb-64e6-460a-80f5-a477cdcd0a58 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:57.938033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74fc3fdb-64e6-460a-80f5-a477cdcd0a58 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:57.959437 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:58.062723 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-24396b0e-e194-4359-875f-b2dab908796a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dcefd7a94ef2407b875e112b37e8116e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:58.078033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 921/1842] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:57 2022] POST /identity/v3/users => generated 284 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:58.083800 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fc3fdb-64e6-460a-80f5-a477cdcd0a58 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:58.084114 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-24396b0e-e194-4359-875f-b2dab908796a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dcefd7a94ef2407b875e112b37e8116e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:58.100478 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fc3fdb-64e6-460a-80f5-a477cdcd0a58 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:58.118931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af316ecd-8d3f-4ec8-a76a-b61902772383 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:58.118931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af316ecd-8d3f-4ec8-a76a-b61902772383 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:58.118931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af316ecd-8d3f-4ec8-a76a-b61902772383 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:58.118931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af316ecd-8d3f-4ec8-a76a-b61902772383 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:58.119390 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af316ecd-8d3f-4ec8-a76a-b61902772383 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:58.119390 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af316ecd-8d3f-4ec8-a76a-b61902772383 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:58.128719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 922/1843] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:58 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:58.132683 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af316ecd-8d3f-4ec8-a76a-b61902772383 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:58.142738 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af316ecd-8d3f-4ec8-a76a-b61902772383 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:58.147594 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70dae6c-0426-4b45-8c9d-0203e9d80007 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:58.148189 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70dae6c-0426-4b45-8c9d-0203e9d80007 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:58.148467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70dae6c-0426-4b45-8c9d-0203e9d80007 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:58.148738 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70dae6c-0426-4b45-8c9d-0203e9d80007 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:58.149447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70dae6c-0426-4b45-8c9d-0203e9d80007 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:58.151056 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70dae6c-0426-4b45-8c9d-0203e9d80007 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:58.161706 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 923/1844] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:58 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:48:58.166742 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70dae6c-0426-4b45-8c9d-0203e9d80007 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:58.176757 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70dae6c-0426-4b45-8c9d-0203e9d80007 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:58.181430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be9bdce-836d-4588-81e5-b7517e905a26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:58.182042 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be9bdce-836d-4588-81e5-b7517e905a26 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:58.182320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be9bdce-836d-4588-81e5-b7517e905a26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:58.182592 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be9bdce-836d-4588-81e5-b7517e905a26 admin admin] PATH_INFO: `/v3/projects/10d20578af134f4fa99d5b3479ef1920/users/7547f6b8678c4f308608f6146ffb5151/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:58.198692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be9bdce-836d-4588-81e5-b7517e905a26 admin admin] RBAC: Authorizing `identity:create_grant(project_id=10d20578af134f4fa99d5b3479ef1920, user_id=7547f6b8678c4f308608f6146ffb5151, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:58.198982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-24396b0e-e194-4359-875f-b2dab908796a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dcefd7a94ef2407b875e112b37e8116e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:58.199623 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-24396b0e-e194-4359-875f-b2dab908796a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dcefd7a94ef2407b875e112b37e8116e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:58.201228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be9bdce-836d-4588-81e5-b7517e905a26 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:58.207808 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-24396b0e-e194-4359-875f-b2dab908796a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dcefd7a94ef2407b875e112b37e8116e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:58.213744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 922/1845] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:48:57 2022] DELETE /identity/v3/users/dcefd7a94ef2407b875e112b37e8116e => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:58.272907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24396b0e-e194-4359-875f-b2dab908796a admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:58.274516 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-24396b0e-e194-4359-875f-b2dab908796a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:58.363171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24396b0e-e194-4359-875f-b2dab908796a admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:58.368456 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33336004-5d4c-40fe-ba73-be8d7b910219 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:58.368891 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33336004-5d4c-40fe-ba73-be8d7b910219 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:58.369173 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33336004-5d4c-40fe-ba73-be8d7b910219 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:58.369447 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33336004-5d4c-40fe-ba73-be8d7b910219 admin admin] PATH_INFO: `/v3/projects/fdcbab241b3b464dbb487a455fbd29f8` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:58.370574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33336004-5d4c-40fe-ba73-be8d7b910219 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fdcbab241b3b464dbb487a455fbd29f8)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:58.372578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33336004-5d4c-40fe-ba73-be8d7b910219 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:58.401861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 924/1846] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:58 2022] PUT /identity/v3/projects/10d20578af134f4fa99d5b3479ef1920/users/7547f6b8678c4f308608f6146ffb5151/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:48:58.413389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5103b3b-0edc-4f47-bb29-8dda21fc9539 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:58.413612 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5103b3b-0edc-4f47-bb29-8dda21fc9539 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:58.413895 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5103b3b-0edc-4f47-bb29-8dda21fc9539 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:58.446267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-c5103b3b-0edc-4f47-bb29-8dda21fc9539 None None] MFA Rules not processed for user `7547f6b8678c4f308608f6146ffb5151`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:58.447436 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c5103b3b-0edc-4f47-bb29-8dda21fc9539 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:58.541186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-33336004-5d4c-40fe-ba73-be8d7b910219 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fdcbab241b3b464dbb487a455fbd29f8', 'request_id': 'req-33336004-5d4c-40fe-ba73-be8d7b910219'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:58.551742 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 925/1847] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:58 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:48:58.557919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:48:58.557919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-33336004-5d4c-40fe-ba73-be8d7b910219 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fdcbab241b3b464dbb487a455fbd29f8', 'request_id': 'req-33336004-5d4c-40fe-ba73-be8d7b910219'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:58.562607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5103b3b-0edc-4f47-bb29-8dda21fc9539 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:58.564411 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c5103b3b-0edc-4f47-bb29-8dda21fc9539 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:58.656888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5103b3b-0edc-4f47-bb29-8dda21fc9539 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:58.687610 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-33336004-5d4c-40fe-ba73-be8d7b910219 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fdcbab241b3b464dbb487a455fbd29f8', 'request_id': 'req-33336004-5d4c-40fe-ba73-be8d7b910219'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:58.699864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-33336004-5d4c-40fe-ba73-be8d7b910219 admin admin] The token cache is being invalidate because project fdcbab241b3b464dbb487a455fbd29f8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:48:58.704277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-33336004-5d4c-40fe-ba73-be8d7b910219 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:58.704277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-33336004-5d4c-40fe-ba73-be8d7b910219 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:58.704277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 923/1848] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:48:58 2022] DELETE /identity/v3/projects/fdcbab241b3b464dbb487a455fbd29f8 => generated 0 bytes in 430 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:58.706634 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33336004-5d4c-40fe-ba73-be8d7b910219 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:58.708047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-33336004-5d4c-40fe-ba73-be8d7b910219 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:58.735083 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41732ff6-74f5-4fc6-acf9-2af0447cb7f4 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ceb9fc31c345ccabaee8f30a3982a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:58.735902 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41732ff6-74f5-4fc6-acf9-2af0447cb7f4 service glance] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:58.736187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41732ff6-74f5-4fc6-acf9-2af0447cb7f4 service glance] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:58.736466 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41732ff6-74f5-4fc6-acf9-2af0447cb7f4 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:58.738217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-41732ff6-74f5-4fc6-acf9-2af0447cb7f4 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:58.784009 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33336004-5d4c-40fe-ba73-be8d7b910219 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:58.790108 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efedd6ac-3a3a-42b1-8318-20b7660f6171 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:58.792867 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efedd6ac-3a3a-42b1-8318-20b7660f6171 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:58.792867 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efedd6ac-3a3a-42b1-8318-20b7660f6171 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:58.792867 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efedd6ac-3a3a-42b1-8318-20b7660f6171 admin admin] PATH_INFO: `/v3/users/57e9eb7c3277426294bcd1ccdb813aba` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:58.792867 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efedd6ac-3a3a-42b1-8318-20b7660f6171 admin admin] RBAC: Authorizing `identity:delete_user(user_id=57e9eb7c3277426294bcd1ccdb813aba)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:58.797810 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efedd6ac-3a3a-42b1-8318-20b7660f6171 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:58.804245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41732ff6-74f5-4fc6-acf9-2af0447cb7f4 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:58.805851 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41732ff6-74f5-4fc6-acf9-2af0447cb7f4 service glance] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:58.824972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 926/1849] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:48:58 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 257 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:48:58.824972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41732ff6-74f5-4fc6-acf9-2af0447cb7f4 service glance] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:58.829950 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-41732ff6-74f5-4fc6-acf9-2af0447cb7f4 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:58.865128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41732ff6-74f5-4fc6-acf9-2af0447cb7f4 service glance] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:58.870440 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77dc6305-388c-4965-9e3d-42c06560a160 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:58.871119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dc6305-388c-4965-9e3d-42c06560a160 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:58.871408 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dc6305-388c-4965-9e3d-42c06560a160 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:58.871690 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dc6305-388c-4965-9e3d-42c06560a160 admin admin] PATH_INFO: `/v3/projects/3002c2f1222949079ed30d3e444c6405` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:58.877127 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77dc6305-388c-4965-9e3d-42c06560a160 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3002c2f1222949079ed30d3e444c6405)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:58.879278 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77dc6305-388c-4965-9e3d-42c06560a160 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:58.996359 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-efedd6ac-3a3a-42b1-8318-20b7660f6171 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '57e9eb7c3277426294bcd1ccdb813aba'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:59.010292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-efedd6ac-3a3a-42b1-8318-20b7660f6171 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '57e9eb7c3277426294bcd1ccdb813aba'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:59.021142 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-77dc6305-388c-4965-9e3d-42c06560a160 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3002c2f1222949079ed30d3e444c6405', 'request_id': 'req-77dc6305-388c-4965-9e3d-42c06560a160'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:59.113515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-efedd6ac-3a3a-42b1-8318-20b7660f6171 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '57e9eb7c3277426294bcd1ccdb813aba'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:59.114308 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-efedd6ac-3a3a-42b1-8318-20b7660f6171 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '57e9eb7c3277426294bcd1ccdb813aba'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:59.122153 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-efedd6ac-3a3a-42b1-8318-20b7660f6171 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '57e9eb7c3277426294bcd1ccdb813aba'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:59.132073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 924/1850] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:48:58 2022] DELETE /identity/v3/users/57e9eb7c3277426294bcd1ccdb813aba => generated 0 bytes in 423 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:48:59.182190 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-77dc6305-388c-4965-9e3d-42c06560a160 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3002c2f1222949079ed30d3e444c6405', 'request_id': 'req-77dc6305-388c-4965-9e3d-42c06560a160'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:59.190833 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-77dc6305-388c-4965-9e3d-42c06560a160 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3002c2f1222949079ed30d3e444c6405', 'request_id': 'req-77dc6305-388c-4965-9e3d-42c06560a160'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:59.205019 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-77dc6305-388c-4965-9e3d-42c06560a160 admin admin] The token cache is being invalidate because project 3002c2f1222949079ed30d3e444c6405 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:48:59.205360 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-77dc6305-388c-4965-9e3d-42c06560a160 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:59.206141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-77dc6305-388c-4965-9e3d-42c06560a160 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:59.212994 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 927/1851] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:48:58 2022] DELETE /identity/v3/projects/3002c2f1222949079ed30d3e444c6405 => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:48:59.229760 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631ad1b2-3753-40fe-a2c1-a5a324ad7b64 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:59.230027 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631ad1b2-3753-40fe-a2c1-a5a324ad7b64 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:59.230412 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631ad1b2-3753-40fe-a2c1-a5a324ad7b64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:59.263197 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-631ad1b2-3753-40fe-a2c1-a5a324ad7b64 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:59.265945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-631ad1b2-3753-40fe-a2c1-a5a324ad7b64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:59.349475 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 925/1852] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:48:59.365744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bd0c12-2a28-404c-aa08-4edab57efbd9 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:59.366035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bd0c12-2a28-404c-aa08-4edab57efbd9 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:59.366314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bd0c12-2a28-404c-aa08-4edab57efbd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:59.386476 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-59bd0c12-2a28-404c-aa08-4edab57efbd9 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:48:59.387735 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-59bd0c12-2a28-404c-aa08-4edab57efbd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:59.396863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 928/1853] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:48:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:48:59.401009 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631ad1b2-3753-40fe-a2c1-a5a324ad7b64 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:59.410793 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631ad1b2-3753-40fe-a2c1-a5a324ad7b64 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:59.415905 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8bc6c0-ceda-4d8e-b680-a8b609f7d33e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:59.416514 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8bc6c0-ceda-4d8e-b680-a8b609f7d33e admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:59.416821 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8bc6c0-ceda-4d8e-b680-a8b609f7d33e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:59.417167 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8bc6c0-ceda-4d8e-b680-a8b609f7d33e admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:59.417817 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8bc6c0-ceda-4d8e-b680-a8b609f7d33e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:48:59.418141 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8bc6c0-ceda-4d8e-b680-a8b609f7d33e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:59.420521 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8bc6c0-ceda-4d8e-b680-a8b609f7d33e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:59.433377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 926/1854] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:48:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:48:59.438361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bd0c12-2a28-404c-aa08-4edab57efbd9 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:59.447914 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bd0c12-2a28-404c-aa08-4edab57efbd9 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:59.455655 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91ad8fe-b0ee-47a4-9b05-bb591800024c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:59.455655 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91ad8fe-b0ee-47a4-9b05-bb591800024c admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:59.455655 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91ad8fe-b0ee-47a4-9b05-bb591800024c admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:59.455655 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91ad8fe-b0ee-47a4-9b05-bb591800024c admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:59.456080 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91ad8fe-b0ee-47a4-9b05-bb591800024c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:59.473037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91ad8fe-b0ee-47a4-9b05-bb591800024c admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:59.569569 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 929/1855] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:48:59 2022] POST /identity/v3/projects => generated 381 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:48:59.573981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8bc6c0-ceda-4d8e-b680-a8b609f7d33e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:59.584732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8bc6c0-ceda-4d8e-b680-a8b609f7d33e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:59.665318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406195b5-7ebd-456c-854c-e269ae742a53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:59.666155 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406195b5-7ebd-456c-854c-e269ae742a53 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:59.666671 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406195b5-7ebd-456c-854c-e269ae742a53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:59.667099 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406195b5-7ebd-456c-854c-e269ae742a53 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:59.667979 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406195b5-7ebd-456c-854c-e269ae742a53 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:59.670060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406195b5-7ebd-456c-854c-e269ae742a53 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:59.724105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91ad8fe-b0ee-47a4-9b05-bb591800024c admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:59.726067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e91ad8fe-b0ee-47a4-9b05-bb591800024c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:48:59.742815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91ad8fe-b0ee-47a4-9b05-bb591800024c admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:59.750542 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962230b8-02f7-488a-8ea4-8bdbdd18296d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:59.751487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962230b8-02f7-488a-8ea4-8bdbdd18296d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:59.751781 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962230b8-02f7-488a-8ea4-8bdbdd18296d admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:59.752051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962230b8-02f7-488a-8ea4-8bdbdd18296d admin admin] PATH_INFO: `/v3/projects/e8f0e60e237c480b8bdd1374d0f8ec53` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:59.753551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962230b8-02f7-488a-8ea4-8bdbdd18296d admin admin] RBAC: Authorizing `identity:delete_project(project_id=e8f0e60e237c480b8bdd1374d0f8ec53)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:59.756058 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962230b8-02f7-488a-8ea4-8bdbdd18296d admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:59.856169 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 927/1856] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:48:59 2022] POST /identity/v3/users => generated 291 bytes in 283 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:48:59.860509 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406195b5-7ebd-456c-854c-e269ae742a53 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:59.870965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406195b5-7ebd-456c-854c-e269ae742a53 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:59.877745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2358b0-5d65-419f-bfbb-b6e71a077a52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:59.878366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2358b0-5d65-419f-bfbb-b6e71a077a52 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:59.878651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2358b0-5d65-419f-bfbb-b6e71a077a52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:59.878979 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2358b0-5d65-419f-bfbb-b6e71a077a52 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:59.879631 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2358b0-5d65-419f-bfbb-b6e71a077a52 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:59.881334 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2358b0-5d65-419f-bfbb-b6e71a077a52 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:59.894934 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 928/1857] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:59 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:59.901586 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2358b0-5d65-419f-bfbb-b6e71a077a52 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:59.914313 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2358b0-5d65-419f-bfbb-b6e71a077a52 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:48:59.919944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fe9462-d03d-4e2d-b1c9-8df7545bd66a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:48:59.920742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fe9462-d03d-4e2d-b1c9-8df7545bd66a admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:48:59.921029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fe9462-d03d-4e2d-b1c9-8df7545bd66a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:48:59.921310 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fe9462-d03d-4e2d-b1c9-8df7545bd66a admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:48:59.921997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fe9462-d03d-4e2d-b1c9-8df7545bd66a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:48:59.927834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fe9462-d03d-4e2d-b1c9-8df7545bd66a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:48:59.942449 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 929/1858] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:48:59 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:48:59.943687 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fe9462-d03d-4e2d-b1c9-8df7545bd66a admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:48:59.956529 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-962230b8-02f7-488a-8ea4-8bdbdd18296d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e8f0e60e237c480b8bdd1374d0f8ec53', 'request_id': 'req-962230b8-02f7-488a-8ea4-8bdbdd18296d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:48:59.958375 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fe9462-d03d-4e2d-b1c9-8df7545bd66a admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:00.030510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db07ecce-d3c3-47b9-ad7e-56f60e7f019c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:00.030510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db07ecce-d3c3-47b9-ad7e-56f60e7f019c admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:00.030859 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db07ecce-d3c3-47b9-ad7e-56f60e7f019c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:00.031005 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db07ecce-d3c3-47b9-ad7e-56f60e7f019c admin admin] PATH_INFO: `/v3/projects/bd2a1d98963c4da89592672d94577c3a/users/d45f3dfd1e464491ad06a9205a957ec9/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:00.048861 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db07ecce-d3c3-47b9-ad7e-56f60e7f019c admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd2a1d98963c4da89592672d94577c3a, user_id=d45f3dfd1e464491ad06a9205a957ec9, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:00.052486 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db07ecce-d3c3-47b9-ad7e-56f60e7f019c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:00.062473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-962230b8-02f7-488a-8ea4-8bdbdd18296d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e8f0e60e237c480b8bdd1374d0f8ec53', 'request_id': 'req-962230b8-02f7-488a-8ea4-8bdbdd18296d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:00.068807 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-962230b8-02f7-488a-8ea4-8bdbdd18296d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e8f0e60e237c480b8bdd1374d0f8ec53', 'request_id': 'req-962230b8-02f7-488a-8ea4-8bdbdd18296d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:00.083186 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-962230b8-02f7-488a-8ea4-8bdbdd18296d admin admin] The token cache is being invalidate because project e8f0e60e237c480b8bdd1374d0f8ec53 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:49:00.083482 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-962230b8-02f7-488a-8ea4-8bdbdd18296d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:00.084344 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-962230b8-02f7-488a-8ea4-8bdbdd18296d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:00.086852 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 930/1859] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:48:59 2022] DELETE /identity/v3/projects/e8f0e60e237c480b8bdd1374d0f8ec53 => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:00.091986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb91bbc2-aab7-4a29-8f1f-dab75e241d2f None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:00.092668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb91bbc2-aab7-4a29-8f1f-dab75e241d2f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:00.093664 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb91bbc2-aab7-4a29-8f1f-dab75e241d2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:00.123392 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-cb91bbc2-aab7-4a29-8f1f-dab75e241d2f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:00.123392 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-cb91bbc2-aab7-4a29-8f1f-dab75e241d2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:00.129918 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 931/1860] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:00.139217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33c335b-ec61-4301-8861-c97dbe854752 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:00.139670 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33c335b-ec61-4301-8861-c97dbe854752 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:00.142662 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33c335b-ec61-4301-8861-c97dbe854752 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:00.148690 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 930/1861] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:48:59 2022] PUT /identity/v3/projects/bd2a1d98963c4da89592672d94577c3a/users/d45f3dfd1e464491ad06a9205a957ec9/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:49:00.160163 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e98f4b-526e-4520-a72c-2199e9ad966d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:00.160540 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e98f4b-526e-4520-a72c-2199e9ad966d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:00.160837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e98f4b-526e-4520-a72c-2199e9ad966d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:00.161008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-d33c335b-ec61-4301-8861-c97dbe854752 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:00.162199 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d33c335b-ec61-4301-8861-c97dbe854752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:00.193984 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-83e98f4b-526e-4520-a72c-2199e9ad966d None None] MFA Rules not processed for user `d45f3dfd1e464491ad06a9205a957ec9`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:00.195362 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-83e98f4b-526e-4520-a72c-2199e9ad966d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:00.231050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 932/1862] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:49:00.233408 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33c335b-ec61-4301-8861-c97dbe854752 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:00.242375 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33c335b-ec61-4301-8861-c97dbe854752 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:00.247094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d70b83-9f3e-4d6b-856a-2b7a9f32fd65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:00.247728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d70b83-9f3e-4d6b-856a-2b7a9f32fd65 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:00.248522 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d70b83-9f3e-4d6b-856a-2b7a9f32fd65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:00.248866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d70b83-9f3e-4d6b-856a-2b7a9f32fd65 admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:00.249856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d70b83-9f3e-4d6b-856a-2b7a9f32fd65 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:49:00.250179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d70b83-9f3e-4d6b-856a-2b7a9f32fd65 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:00.252134 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:00.253950 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d70b83-9f3e-4d6b-856a-2b7a9f32fd65 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:00.267781 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 933/1863] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:49:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:49:00.274635 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 931/1864] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:00 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:49:00.276078 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d70b83-9f3e-4d6b-856a-2b7a9f32fd65 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:00.277678 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e98f4b-526e-4520-a72c-2199e9ad966d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:00.279517 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-83e98f4b-526e-4520-a72c-2199e9ad966d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:00.286368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d70b83-9f3e-4d6b-856a-2b7a9f32fd65 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:00.291144 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8dc7d40-03df-4c1a-b323-683bd219ca4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:00.291762 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dc7d40-03df-4c1a-b323-683bd219ca4f admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:00.292044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dc7d40-03df-4c1a-b323-683bd219ca4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:00.292320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dc7d40-03df-4c1a-b323-683bd219ca4f admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:00.292952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8dc7d40-03df-4c1a-b323-683bd219ca4f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:00.295218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e98f4b-526e-4520-a72c-2199e9ad966d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:00.295472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8dc7d40-03df-4c1a-b323-683bd219ca4f admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:00.300361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea9a332-4e60-481b-9431-5bbd6a611d12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:00.300986 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea9a332-4e60-481b-9431-5bbd6a611d12 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:00.301265 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea9a332-4e60-481b-9431-5bbd6a611d12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:00.303554 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea9a332-4e60-481b-9431-5bbd6a611d12 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:00.303554 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea9a332-4e60-481b-9431-5bbd6a611d12 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:00.304378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea9a332-4e60-481b-9431-5bbd6a611d12 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:00.422544 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 934/1865] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:49:00 2022] POST /identity/v3/projects => generated 391 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:49:00.425909 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8dc7d40-03df-4c1a-b323-683bd219ca4f admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:00.450171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 932/1866] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:49:00 2022] POST /identity/v3/projects => generated 383 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:49:00.454621 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea9a332-4e60-481b-9431-5bbd6a611d12 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:00.538570 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8dc7d40-03df-4c1a-b323-683bd219ca4f admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:00.567024 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea9a332-4e60-481b-9431-5bbd6a611d12 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:00.572983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e118eb14-66d7-45e9-821b-7be8afe125c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:00.574186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e118eb14-66d7-45e9-821b-7be8afe125c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:00.574481 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e118eb14-66d7-45e9-821b-7be8afe125c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:00.574745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e118eb14-66d7-45e9-821b-7be8afe125c8 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:00.575557 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e118eb14-66d7-45e9-821b-7be8afe125c8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:00.578515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e118eb14-66d7-45e9-821b-7be8afe125c8 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:00.623707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1454b0d2-e13d-4a55-b7c3-b0574a3227e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:00.624634 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1454b0d2-e13d-4a55-b7c3-b0574a3227e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:00.625060 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1454b0d2-e13d-4a55-b7c3-b0574a3227e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:00.625444 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1454b0d2-e13d-4a55-b7c3-b0574a3227e5 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:00.626233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1454b0d2-e13d-4a55-b7c3-b0574a3227e5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:00.628643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1454b0d2-e13d-4a55-b7c3-b0574a3227e5 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:00.674311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 933/1867] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:49:00 2022] POST /identity/v3/users => generated 298 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:49:00.678439 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e118eb14-66d7-45e9-821b-7be8afe125c8 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:00.689250 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e118eb14-66d7-45e9-821b-7be8afe125c8 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:00.694042 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530cb339-3a7c-4475-a9a8-6c73d47a9f35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:00.694653 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530cb339-3a7c-4475-a9a8-6c73d47a9f35 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:00.694992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530cb339-3a7c-4475-a9a8-6c73d47a9f35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:00.695274 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530cb339-3a7c-4475-a9a8-6c73d47a9f35 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:00.695920 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530cb339-3a7c-4475-a9a8-6c73d47a9f35 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:00.697514 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530cb339-3a7c-4475-a9a8-6c73d47a9f35 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:00.710720 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 934/1868] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:00 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:49:00.714738 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530cb339-3a7c-4475-a9a8-6c73d47a9f35 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:00.725625 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530cb339-3a7c-4475-a9a8-6c73d47a9f35 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:00.731015 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21143a8-60ae-4486-be80-de1387225a90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:00.731776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21143a8-60ae-4486-be80-de1387225a90 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:00.732212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21143a8-60ae-4486-be80-de1387225a90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:00.732484 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21143a8-60ae-4486-be80-de1387225a90 admin admin] PATH_INFO: `/v3/projects/b11a7df3ff2d4b36a9e38fad1de5dd45/users/03eb349a23c547a4b0565ae32603fdef/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:00.742581 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 935/1869] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:49:00 2022] POST /identity/v3/users => generated 296 bytes in 317 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:49:00.754192 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1454b0d2-e13d-4a55-b7c3-b0574a3227e5 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:00.756649 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21143a8-60ae-4486-be80-de1387225a90 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b11a7df3ff2d4b36a9e38fad1de5dd45, user_id=03eb349a23c547a4b0565ae32603fdef, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:00.761026 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21143a8-60ae-4486-be80-de1387225a90 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:00.764075 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1454b0d2-e13d-4a55-b7c3-b0574a3227e5 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:00.771064 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed54024b-bfd2-4fce-8fbc-cf3b2bb91c15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:00.772072 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed54024b-bfd2-4fce-8fbc-cf3b2bb91c15 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:00.772431 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed54024b-bfd2-4fce-8fbc-cf3b2bb91c15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:00.772779 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed54024b-bfd2-4fce-8fbc-cf3b2bb91c15 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:00.773657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed54024b-bfd2-4fce-8fbc-cf3b2bb91c15 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:00.775605 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed54024b-bfd2-4fce-8fbc-cf3b2bb91c15 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:00.787583 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 936/1870] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:00 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:00.791645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed54024b-bfd2-4fce-8fbc-cf3b2bb91c15 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:00.802413 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed54024b-bfd2-4fce-8fbc-cf3b2bb91c15 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:00.807202 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db17b92-1d74-420c-b5a4-a8af2560dc2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:00.807820 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db17b92-1d74-420c-b5a4-a8af2560dc2e admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:00.808233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db17b92-1d74-420c-b5a4-a8af2560dc2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:00.808518 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db17b92-1d74-420c-b5a4-a8af2560dc2e admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:00.809155 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db17b92-1d74-420c-b5a4-a8af2560dc2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:00.811337 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db17b92-1d74-420c-b5a4-a8af2560dc2e admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:00.822967 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 937/1871] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:00 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:49:00.827856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db17b92-1d74-420c-b5a4-a8af2560dc2e admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:00.839044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db17b92-1d74-420c-b5a4-a8af2560dc2e admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:00.847249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fb84d4-b2bb-461f-b8bc-130c04c88a8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:00.848696 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fb84d4-b2bb-461f-b8bc-130c04c88a8e admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:00.849071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fb84d4-b2bb-461f-b8bc-130c04c88a8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:00.849501 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fb84d4-b2bb-461f-b8bc-130c04c88a8e admin admin] PATH_INFO: `/v3/projects/1bd01cf0fc12498e9b594dd83c23a646/users/e0e8baca193c4c89b1f231ba002a91ed/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:00.857126 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 935/1872] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:00 2022] PUT /identity/v3/projects/b11a7df3ff2d4b36a9e38fad1de5dd45/users/03eb349a23c547a4b0565ae32603fdef/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:49:00.862076 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21143a8-60ae-4486-be80-de1387225a90 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:00.869968 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fb84d4-b2bb-461f-b8bc-130c04c88a8e admin admin] RBAC: Authorizing `identity:create_grant(project_id=1bd01cf0fc12498e9b594dd83c23a646, user_id=e0e8baca193c4c89b1f231ba002a91ed, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:00.873194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fb84d4-b2bb-461f-b8bc-130c04c88a8e admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:00.930241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21143a8-60ae-4486-be80-de1387225a90 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:00.935503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378bf777-a4b5-464f-b375-665c0577f5f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:00.936143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378bf777-a4b5-464f-b375-665c0577f5f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:00.936425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378bf777-a4b5-464f-b375-665c0577f5f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:00.936733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378bf777-a4b5-464f-b375-665c0577f5f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:00.937325 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378bf777-a4b5-464f-b375-665c0577f5f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:00.938936 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378bf777-a4b5-464f-b375-665c0577f5f5 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:00.953883 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 936/1873] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:00 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:49:00.958252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378bf777-a4b5-464f-b375-665c0577f5f5 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:00.965386 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 938/1874] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:00 2022] PUT /identity/v3/projects/1bd01cf0fc12498e9b594dd83c23a646/users/e0e8baca193c4c89b1f231ba002a91ed/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:49:00.970389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378bf777-a4b5-464f-b375-665c0577f5f5 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:00.981258 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed4c17d-5757-4440-9af0-2267066febe1 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:00.981258 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed4c17d-5757-4440-9af0-2267066febe1 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:00.981258 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed4c17d-5757-4440-9af0-2267066febe1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:01.008729 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-1ed4c17d-5757-4440-9af0-2267066febe1 None None] MFA Rules not processed for user `e0e8baca193c4c89b1f231ba002a91ed`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:01.010264 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1ed4c17d-5757-4440-9af0-2267066febe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:01.019272 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:01.036919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b52d6a-a02c-428c-a903-06075e89a960 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:01.037843 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b52d6a-a02c-428c-a903-06075e89a960 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:01.038474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b52d6a-a02c-428c-a903-06075e89a960 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:01.038474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b52d6a-a02c-428c-a903-06075e89a960 admin admin] PATH_INFO: `/v3/projects/b11a7df3ff2d4b36a9e38fad1de5dd45/users/03eb349a23c547a4b0565ae32603fdef/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:01.043982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b52d6a-a02c-428c-a903-06075e89a960 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b11a7df3ff2d4b36a9e38fad1de5dd45, user_id=03eb349a23c547a4b0565ae32603fdef, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:01.044820 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b52d6a-a02c-428c-a903-06075e89a960 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:01.057598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 937/1875] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:00 2022] PUT /identity/v3/projects/b11a7df3ff2d4b36a9e38fad1de5dd45/users/03eb349a23c547a4b0565ae32603fdef/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:01.070006 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebccdcea-f9fd-4b35-b879-72189ae51b87 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:01.070339 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebccdcea-f9fd-4b35-b879-72189ae51b87 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:01.072807 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebccdcea-f9fd-4b35-b879-72189ae51b87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:01.101455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 939/1876] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:00 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:49:01.102511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed4c17d-5757-4440-9af0-2267066febe1 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:01.109030 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ebccdcea-f9fd-4b35-b879-72189ae51b87 None None] MFA Rules not processed for user `03eb349a23c547a4b0565ae32603fdef`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:01.110560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ebccdcea-f9fd-4b35-b879-72189ae51b87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:01.113029 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed4c17d-5757-4440-9af0-2267066febe1 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:01.214926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348e9afc-765c-4594-8200-bd727211ce85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:01.215631 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348e9afc-765c-4594-8200-bd727211ce85 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:01.215857 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348e9afc-765c-4594-8200-bd727211ce85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:01.216134 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348e9afc-765c-4594-8200-bd727211ce85 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:01.217351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348e9afc-765c-4594-8200-bd727211ce85 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:01.220703 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348e9afc-765c-4594-8200-bd727211ce85 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:01.240269 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 938/1877] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:01 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:49:01.255613 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebccdcea-f9fd-4b35-b879-72189ae51b87 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:01.258920 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ebccdcea-f9fd-4b35-b879-72189ae51b87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:01.341049 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebccdcea-f9fd-4b35-b879-72189ae51b87 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:01.355162 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b166945f-e06d-4579-ada8-9cdc2e81f1f7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:01.355162 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b166945f-e06d-4579-ada8-9cdc2e81f1f7 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:01.355162 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b166945f-e06d-4579-ada8-9cdc2e81f1f7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:01.355162 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b166945f-e06d-4579-ada8-9cdc2e81f1f7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:01.362269 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b166945f-e06d-4579-ada8-9cdc2e81f1f7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:01.363762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b166945f-e06d-4579-ada8-9cdc2e81f1f7 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:01.374488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 939/1878] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:01 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:49:01.403352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 940/1879] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:49:01 2022] POST /identity/v3/projects => generated 391 bytes in 298 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:49:01.405251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b166945f-e06d-4579-ada8-9cdc2e81f1f7 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:01.420489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b166945f-e06d-4579-ada8-9cdc2e81f1f7 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:01.519849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c27e7b-724f-4693-9a16-f970c88543e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:01.520624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c27e7b-724f-4693-9a16-f970c88543e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:01.520793 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c27e7b-724f-4693-9a16-f970c88543e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:01.521069 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c27e7b-724f-4693-9a16-f970c88543e9 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:01.521773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c27e7b-724f-4693-9a16-f970c88543e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:01.523789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c27e7b-724f-4693-9a16-f970c88543e9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:01.594853 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348e9afc-765c-4594-8200-bd727211ce85 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:01.689081 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348e9afc-765c-4594-8200-bd727211ce85 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:01.694204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bcff11-a497-4184-8802-58ccda42bd37 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:01.712960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bcff11-a497-4184-8802-58ccda42bd37 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:01.712960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bcff11-a497-4184-8802-58ccda42bd37 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:01.712960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bcff11-a497-4184-8802-58ccda42bd37 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:01.713288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 940/1880] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:49:01 2022] POST /identity/v3/users => generated 302 bytes in 293 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:01.713288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c27e7b-724f-4693-9a16-f970c88543e9 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:01.726956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c27e7b-724f-4693-9a16-f970c88543e9 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:01.738127 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9ccb0b-14d6-4ab6-8d7f-ec78a01a2158 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:01.738834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9ccb0b-14d6-4ab6-8d7f-ec78a01a2158 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:01.739584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9ccb0b-14d6-4ab6-8d7f-ec78a01a2158 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:01.739584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9ccb0b-14d6-4ab6-8d7f-ec78a01a2158 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:01.739962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9ccb0b-14d6-4ab6-8d7f-ec78a01a2158 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:01.741599 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9ccb0b-14d6-4ab6-8d7f-ec78a01a2158 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:01.773241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 941/1881] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:01 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:49:01.779134 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9ccb0b-14d6-4ab6-8d7f-ec78a01a2158 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:01.817557 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9ccb0b-14d6-4ab6-8d7f-ec78a01a2158 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:01.832092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702bd3df-e733-4908-856f-d12838a2ba96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:01.840742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702bd3df-e733-4908-856f-d12838a2ba96 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:01.840742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702bd3df-e733-4908-856f-d12838a2ba96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:01.844779 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702bd3df-e733-4908-856f-d12838a2ba96 admin admin] PATH_INFO: `/v3/projects/943f0e1e164148bab3612e84cf2891a6/users/80c28c89b7734c74ae32dcf41e88f432/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:01.868303 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bcff11-a497-4184-8802-58ccda42bd37 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:01.879038 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bcff11-a497-4184-8802-58ccda42bd37 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:01.889688 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702bd3df-e733-4908-856f-d12838a2ba96 admin admin] RBAC: Authorizing `identity:create_grant(project_id=943f0e1e164148bab3612e84cf2891a6, user_id=80c28c89b7734c74ae32dcf41e88f432, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:01.893074 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702bd3df-e733-4908-856f-d12838a2ba96 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:01.901483 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 941/1882] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:01 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:49:02.014992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 942/1883] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:01 2022] PUT /identity/v3/projects/943f0e1e164148bab3612e84cf2891a6/users/80c28c89b7734c74ae32dcf41e88f432/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:02.016727 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bcff11-a497-4184-8802-58ccda42bd37 service cinder] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:02.036333 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bcff11-a497-4184-8802-58ccda42bd37 service cinder] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:02.061762 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:02.159039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00e77c5-413a-4167-9f68-c2e8aa80f4a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:02.161146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00e77c5-413a-4167-9f68-c2e8aa80f4a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:02.161950 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00e77c5-413a-4167-9f68-c2e8aa80f4a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:02.162510 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00e77c5-413a-4167-9f68-c2e8aa80f4a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:02.163401 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00e77c5-413a-4167-9f68-c2e8aa80f4a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:02.166248 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00e77c5-413a-4167-9f68-c2e8aa80f4a5 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:02.206522 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702bd3df-e733-4908-856f-d12838a2ba96 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:02.213741 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 942/1884] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:02 2022] GET /identity/v3/roles => generated 1420 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:02.219666 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702bd3df-e733-4908-856f-d12838a2ba96 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:02.224848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1945c6e-2db5-4908-81b6-b47f8788b95e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:02.226349 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1945c6e-2db5-4908-81b6-b47f8788b95e admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:02.226349 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1945c6e-2db5-4908-81b6-b47f8788b95e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:02.226541 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1945c6e-2db5-4908-81b6-b47f8788b95e admin admin] PATH_INFO: `/v3/projects/943f0e1e164148bab3612e84cf2891a6/users/80c28c89b7734c74ae32dcf41e88f432/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:02.229395 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1945c6e-2db5-4908-81b6-b47f8788b95e admin admin] RBAC: Authorizing `identity:create_grant(project_id=943f0e1e164148bab3612e84cf2891a6, user_id=80c28c89b7734c74ae32dcf41e88f432, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:02.232835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1945c6e-2db5-4908-81b6-b47f8788b95e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:02.250797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 943/1885] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:02 2022] PUT /identity/v3/projects/943f0e1e164148bab3612e84cf2891a6/users/80c28c89b7734c74ae32dcf41e88f432/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:02.260520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c185fcc-2a26-4995-8f91-aa8834742e3d None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:02.260888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c185fcc-2a26-4995-8f91-aa8834742e3d None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:02.261262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c185fcc-2a26-4995-8f91-aa8834742e3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:02.320543 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-7c185fcc-2a26-4995-8f91-aa8834742e3d None None] MFA Rules not processed for user `80c28c89b7734c74ae32dcf41e88f432`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:02.321886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-7c185fcc-2a26-4995-8f91-aa8834742e3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:02.455371 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 943/1886] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:02 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:02.464769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1945c6e-2db5-4908-81b6-b47f8788b95e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:02.560720 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:02.565389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1945c6e-2db5-4908-81b6-b47f8788b95e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:02.582377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478519f9-0ef2-4aad-b76c-29528ce33570 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:02.583054 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478519f9-0ef2-4aad-b76c-29528ce33570 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:02.583343 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478519f9-0ef2-4aad-b76c-29528ce33570 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:02.583624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478519f9-0ef2-4aad-b76c-29528ce33570 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:02.607567 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478519f9-0ef2-4aad-b76c-29528ce33570 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:02.610826 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478519f9-0ef2-4aad-b76c-29528ce33570 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:02.615196 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c185fcc-2a26-4995-8f91-aa8834742e3d None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:02.622135 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 944/1887] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:02 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:02.690731 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478519f9-0ef2-4aad-b76c-29528ce33570 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:02.692673 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-478519f9-0ef2-4aad-b76c-29528ce33570 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:02.712315 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478519f9-0ef2-4aad-b76c-29528ce33570 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:02.719278 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711263d5-4ec6-484f-9202-b869dc2208f3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:02.720201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711263d5-4ec6-484f-9202-b869dc2208f3 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:02.720646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711263d5-4ec6-484f-9202-b869dc2208f3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:02.721018 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711263d5-4ec6-484f-9202-b869dc2208f3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:02.727078 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c185fcc-2a26-4995-8f91-aa8834742e3d None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:02.738107 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19142b66-58f2-4425-b269-cc173c7cd7db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:02.739458 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19142b66-58f2-4425-b269-cc173c7cd7db admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:02.739812 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19142b66-58f2-4425-b269-cc173c7cd7db admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:02.740099 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19142b66-58f2-4425-b269-cc173c7cd7db admin admin] PATH_INFO: `/v3/users/d45f3dfd1e464491ad06a9205a957ec9` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:02.748399 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19142b66-58f2-4425-b269-cc173c7cd7db admin admin] RBAC: Authorizing `identity:delete_user(user_id=d45f3dfd1e464491ad06a9205a957ec9)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:02.750392 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19142b66-58f2-4425-b269-cc173c7cd7db admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:02.777386 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:02.811813 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711263d5-4ec6-484f-9202-b869dc2208f3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:02.813253 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711263d5-4ec6-484f-9202-b869dc2208f3 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:02.824061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 945/1888] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:02 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:49:03.018681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-19142b66-58f2-4425-b269-cc173c7cd7db admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd45f3dfd1e464491ad06a9205a957ec9'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:03.124743 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-19142b66-58f2-4425-b269-cc173c7cd7db admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd45f3dfd1e464491ad06a9205a957ec9'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:03.137870 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-19142b66-58f2-4425-b269-cc173c7cd7db admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd45f3dfd1e464491ad06a9205a957ec9'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:03.138547 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-19142b66-58f2-4425-b269-cc173c7cd7db admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd45f3dfd1e464491ad06a9205a957ec9'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:03.145701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-19142b66-58f2-4425-b269-cc173c7cd7db admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd45f3dfd1e464491ad06a9205a957ec9'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:03.158693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 944/1889] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:49:02 2022] DELETE /identity/v3/users/d45f3dfd1e464491ad06a9205a957ec9 => generated 0 bytes in 544 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:03.163637 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b66379f-8b5f-4cc5-ba93-a3e3d931db50 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:03.164103 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b66379f-8b5f-4cc5-ba93-a3e3d931db50 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:03.164388 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b66379f-8b5f-4cc5-ba93-a3e3d931db50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:03.184457 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7b66379f-8b5f-4cc5-ba93-a3e3d931db50 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:03.185762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7b66379f-8b5f-4cc5-ba93-a3e3d931db50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:03.258364 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 946/1890] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:03.266551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19142b66-58f2-4425-b269-cc173c7cd7db admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:03.268322 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-19142b66-58f2-4425-b269-cc173c7cd7db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:03.338863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19142b66-58f2-4425-b269-cc173c7cd7db admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:03.343026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b883675b-6ff9-4c33-831a-27819bed840d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:03.343848 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b883675b-6ff9-4c33-831a-27819bed840d service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:03.344284 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b883675b-6ff9-4c33-831a-27819bed840d service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:03.346407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b883675b-6ff9-4c33-831a-27819bed840d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:03.354530 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b883675b-6ff9-4c33-831a-27819bed840d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:03.356191 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b883675b-6ff9-4c33-831a-27819bed840d service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:03.369384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 945/1891] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:04.158260 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3f0e0d-fde6-4ec5-91ae-8a6a4a463092 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:04.158824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3f0e0d-fde6-4ec5-91ae-8a6a4a463092 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:04.158824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3f0e0d-fde6-4ec5-91ae-8a6a4a463092 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:04.161171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 947/1892] 10.222.0.49 () {54 vars in 921 bytes} [Fri Sep 16 03:49:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:49:04.170545 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dad8bc3-d8e7-4124-b4b8-4ac2f71a9012 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:04.171061 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dad8bc3-d8e7-4124-b4b8-4ac2f71a9012 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:04.171217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dad8bc3-d8e7-4124-b4b8-4ac2f71a9012 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:04.195773 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-6dad8bc3-d8e7-4124-b4b8-4ac2f71a9012 None None] MFA Rules not processed for user `ded1bcd10e62499a9aaa0ece5aa55c80`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:04.197016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-6dad8bc3-d8e7-4124-b4b8-4ac2f71a9012 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:04.200323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:04.245712 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3f0e0d-fde6-4ec5-91ae-8a6a4a463092 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:04.247610 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fc3f0e0d-fde6-4ec5-91ae-8a6a4a463092 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:04.270894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3f0e0d-fde6-4ec5-91ae-8a6a4a463092 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:04.276937 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f32ed9-7e3f-4070-84cf-3666b7f6977a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:04.280792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f32ed9-7e3f-4070-84cf-3666b7f6977a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:04.280792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f32ed9-7e3f-4070-84cf-3666b7f6977a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:04.280792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f32ed9-7e3f-4070-84cf-3666b7f6977a admin admin] PATH_INFO: `/v3/projects/bd2a1d98963c4da89592672d94577c3a` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:04.281133 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f32ed9-7e3f-4070-84cf-3666b7f6977a admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd2a1d98963c4da89592672d94577c3a)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:04.283350 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f32ed9-7e3f-4070-84cf-3666b7f6977a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:04.333914 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 946/1893] 10.222.0.49 () {58 vars in 1018 bytes} [Fri Sep 16 03:49:04 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:04.352687 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dad8bc3-d8e7-4124-b4b8-4ac2f71a9012 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:04.353812 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-6dad8bc3-d8e7-4124-b4b8-4ac2f71a9012 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:04.370885 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dad8bc3-d8e7-4124-b4b8-4ac2f71a9012 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:04.377784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2c37c6-0b59-4f90-8606-4a2f611f795b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:04.378419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2c37c6-0b59-4f90-8606-4a2f611f795b service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:04.378705 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2c37c6-0b59-4f90-8606-4a2f611f795b service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:04.379007 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2c37c6-0b59-4f90-8606-4a2f611f795b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:04.388472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2c37c6-0b59-4f90-8606-4a2f611f795b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:04.390939 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2c37c6-0b59-4f90-8606-4a2f611f795b service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:04.403107 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 947/1894] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:04.470932 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d0f32ed9-7e3f-4070-84cf-3666b7f6977a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd2a1d98963c4da89592672d94577c3a', 'request_id': 'req-d0f32ed9-7e3f-4070-84cf-3666b7f6977a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:04.487142 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d0f32ed9-7e3f-4070-84cf-3666b7f6977a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd2a1d98963c4da89592672d94577c3a', 'request_id': 'req-d0f32ed9-7e3f-4070-84cf-3666b7f6977a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:04.581925 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d0f32ed9-7e3f-4070-84cf-3666b7f6977a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd2a1d98963c4da89592672d94577c3a', 'request_id': 'req-d0f32ed9-7e3f-4070-84cf-3666b7f6977a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:04.606980 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d0f32ed9-7e3f-4070-84cf-3666b7f6977a admin admin] The token cache is being invalidate because project bd2a1d98963c4da89592672d94577c3a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:49:04.607989 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d0f32ed9-7e3f-4070-84cf-3666b7f6977a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:04.608899 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d0f32ed9-7e3f-4070-84cf-3666b7f6977a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:04.612196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 948/1895] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:49:04 2022] DELETE /identity/v3/projects/bd2a1d98963c4da89592672d94577c3a => generated 0 bytes in 367 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:49:04.616598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2c37c6-0b59-4f90-8606-4a2f611f795b service nova] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:04.630605 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2c37c6-0b59-4f90-8606-4a2f611f795b service nova] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:04.741610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b68eaf-fba9-44f2-9861-2ce9b1ac47f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:04.742410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b68eaf-fba9-44f2-9861-2ce9b1ac47f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:04.742813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b68eaf-fba9-44f2-9861-2ce9b1ac47f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:04.743199 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b68eaf-fba9-44f2-9861-2ce9b1ac47f6 admin admin] PATH_INFO: `/v3/users/03eb349a23c547a4b0565ae32603fdef` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:04.744566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b68eaf-fba9-44f2-9861-2ce9b1ac47f6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=03eb349a23c547a4b0565ae32603fdef)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:04.746752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b68eaf-fba9-44f2-9861-2ce9b1ac47f6 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:05.016345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-82b68eaf-fba9-44f2-9861-2ce9b1ac47f6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '03eb349a23c547a4b0565ae32603fdef'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:05.113685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-82b68eaf-fba9-44f2-9861-2ce9b1ac47f6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '03eb349a23c547a4b0565ae32603fdef'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:05.142430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:05.151799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-82b68eaf-fba9-44f2-9861-2ce9b1ac47f6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '03eb349a23c547a4b0565ae32603fdef'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:05.162154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-82b68eaf-fba9-44f2-9861-2ce9b1ac47f6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '03eb349a23c547a4b0565ae32603fdef'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:05.178696 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-82b68eaf-fba9-44f2-9861-2ce9b1ac47f6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '03eb349a23c547a4b0565ae32603fdef'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:05.189108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 948/1896] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:49:04 2022] DELETE /identity/v3/users/03eb349a23c547a4b0565ae32603fdef => generated 0 bytes in 574 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:49:05.760425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:05.933290 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f32ed9-7e3f-4070-84cf-3666b7f6977a admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:05.935000 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d0f32ed9-7e3f-4070-84cf-3666b7f6977a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:05.951457 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f32ed9-7e3f-4070-84cf-3666b7f6977a admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:06.037566 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a514017-b29a-481e-92e0-3a8a61d2a4e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:06.038164 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a514017-b29a-481e-92e0-3a8a61d2a4e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:06.038608 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a514017-b29a-481e-92e0-3a8a61d2a4e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:06.043181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a514017-b29a-481e-92e0-3a8a61d2a4e0 admin admin] PATH_INFO: `/v3/projects/b11a7df3ff2d4b36a9e38fad1de5dd45` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:06.043181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a514017-b29a-481e-92e0-3a8a61d2a4e0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b11a7df3ff2d4b36a9e38fad1de5dd45)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:06.043181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a514017-b29a-481e-92e0-3a8a61d2a4e0 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:06.162724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1a514017-b29a-481e-92e0-3a8a61d2a4e0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b11a7df3ff2d4b36a9e38fad1de5dd45', 'request_id': 'req-1a514017-b29a-481e-92e0-3a8a61d2a4e0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:06.170679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1a514017-b29a-481e-92e0-3a8a61d2a4e0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b11a7df3ff2d4b36a9e38fad1de5dd45', 'request_id': 'req-1a514017-b29a-481e-92e0-3a8a61d2a4e0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:06.331120 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1a514017-b29a-481e-92e0-3a8a61d2a4e0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b11a7df3ff2d4b36a9e38fad1de5dd45', 'request_id': 'req-1a514017-b29a-481e-92e0-3a8a61d2a4e0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:06.351457 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1a514017-b29a-481e-92e0-3a8a61d2a4e0 admin admin] The token cache is being invalidate because project b11a7df3ff2d4b36a9e38fad1de5dd45 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:49:06.351758 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1a514017-b29a-481e-92e0-3a8a61d2a4e0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:06.352634 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1a514017-b29a-481e-92e0-3a8a61d2a4e0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:06.354362 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 949/1897] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:49:05 2022] DELETE /identity/v3/projects/b11a7df3ff2d4b36a9e38fad1de5dd45 => generated 0 bytes in 421 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:06.359775 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0663b279-a889-42a6-ab80-64176c6f0e1c None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:06.360251 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0663b279-a889-42a6-ab80-64176c6f0e1c None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:06.360340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0663b279-a889-42a6-ab80-64176c6f0e1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:06.384632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-0663b279-a889-42a6-ab80-64176c6f0e1c None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:06.387500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-0663b279-a889-42a6-ab80-64176c6f0e1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:06.464778 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 949/1898] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:06.473528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c12fd2-c09e-4e5a-b596-f3f0ffdcfcc8 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:06.473799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c12fd2-c09e-4e5a-b596-f3f0ffdcfcc8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:06.474203 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c12fd2-c09e-4e5a-b596-f3f0ffdcfcc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:06.503005 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-17c12fd2-c09e-4e5a-b596-f3f0ffdcfcc8 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:06.504166 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-17c12fd2-c09e-4e5a-b596-f3f0ffdcfcc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:06.516389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 950/1899] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:49:06.518116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0663b279-a889-42a6-ab80-64176c6f0e1c None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:06.534137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0663b279-a889-42a6-ab80-64176c6f0e1c None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:06.540920 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f8b536-bdc6-4d6a-8857-9e09167d537a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:06.546892 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f8b536-bdc6-4d6a-8857-9e09167d537a admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:06.546892 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f8b536-bdc6-4d6a-8857-9e09167d537a admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:06.546892 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f8b536-bdc6-4d6a-8857-9e09167d537a admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:06.546892 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f8b536-bdc6-4d6a-8857-9e09167d537a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:49:06.546892 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f8b536-bdc6-4d6a-8857-9e09167d537a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:06.546892 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f8b536-bdc6-4d6a-8857-9e09167d537a admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:06.564732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 950/1900] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:49:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:49:06.566136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c12fd2-c09e-4e5a-b596-f3f0ffdcfcc8 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:06.575159 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:06.583164 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c12fd2-c09e-4e5a-b596-f3f0ffdcfcc8 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:06.588147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0996efb7-cbd5-4302-bba1-b8e392af2853 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:06.588852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0996efb7-cbd5-4302-bba1-b8e392af2853 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:06.589212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0996efb7-cbd5-4302-bba1-b8e392af2853 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:06.589703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0996efb7-cbd5-4302-bba1-b8e392af2853 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:06.590431 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0996efb7-cbd5-4302-bba1-b8e392af2853 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:06.595049 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0996efb7-cbd5-4302-bba1-b8e392af2853 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:06.735272 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 951/1901] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:49:06 2022] POST /identity/v3/projects => generated 373 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:49:06.739439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f8b536-bdc6-4d6a-8857-9e09167d537a admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:06.749765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f8b536-bdc6-4d6a-8857-9e09167d537a admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:06.822374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d60dc1e-43bb-43e3-a2b7-6fae2166bf22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:06.823348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d60dc1e-43bb-43e3-a2b7-6fae2166bf22 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:06.823779 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d60dc1e-43bb-43e3-a2b7-6fae2166bf22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:06.824314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d60dc1e-43bb-43e3-a2b7-6fae2166bf22 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:06.825024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d60dc1e-43bb-43e3-a2b7-6fae2166bf22 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:06.827793 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d60dc1e-43bb-43e3-a2b7-6fae2166bf22 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:06.947365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 951/1902] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:49:06 2022] POST /identity/v3/users => generated 287 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:06.950867 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0996efb7-cbd5-4302-bba1-b8e392af2853 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:06.978740 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0996efb7-cbd5-4302-bba1-b8e392af2853 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:06.985746 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ac00f7-c7c2-4513-ac41-64a6309c49d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:06.986819 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ac00f7-c7c2-4513-ac41-64a6309c49d5 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:06.987234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ac00f7-c7c2-4513-ac41-64a6309c49d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:06.987607 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ac00f7-c7c2-4513-ac41-64a6309c49d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:06.988354 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ac00f7-c7c2-4513-ac41-64a6309c49d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:06.990052 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ac00f7-c7c2-4513-ac41-64a6309c49d5 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:07.001843 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 952/1903] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:06 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:07.005772 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d60dc1e-43bb-43e3-a2b7-6fae2166bf22 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:07.023427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d60dc1e-43bb-43e3-a2b7-6fae2166bf22 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:07.029832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88e7892-6263-4078-a63a-fed4157285ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:07.030586 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88e7892-6263-4078-a63a-fed4157285ad admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:07.031024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88e7892-6263-4078-a63a-fed4157285ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:07.031450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88e7892-6263-4078-a63a-fed4157285ad admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:07.032491 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88e7892-6263-4078-a63a-fed4157285ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:07.034560 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88e7892-6263-4078-a63a-fed4157285ad admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:07.047560 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 952/1904] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:07 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:07.051921 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ac00f7-c7c2-4513-ac41-64a6309c49d5 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:07.062530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ac00f7-c7c2-4513-ac41-64a6309c49d5 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:07.068120 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725ffeb9-c2a2-41be-9bfd-dc56e768428d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:07.068872 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725ffeb9-c2a2-41be-9bfd-dc56e768428d admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:07.069226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725ffeb9-c2a2-41be-9bfd-dc56e768428d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:07.069566 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725ffeb9-c2a2-41be-9bfd-dc56e768428d admin admin] PATH_INFO: `/v3/projects/bed14ff70a404f71a3e7eb574ea3a8cb/users/bc5eb4c34a614860921f281fc7d4413c/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:07.089598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725ffeb9-c2a2-41be-9bfd-dc56e768428d admin admin] RBAC: Authorizing `identity:create_grant(project_id=bed14ff70a404f71a3e7eb574ea3a8cb, user_id=bc5eb4c34a614860921f281fc7d4413c, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:07.092730 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725ffeb9-c2a2-41be-9bfd-dc56e768428d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:07.196815 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 953/1905] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:07 2022] PUT /identity/v3/projects/bed14ff70a404f71a3e7eb574ea3a8cb/users/bc5eb4c34a614860921f281fc7d4413c/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:49:07.202466 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83871e8d-8d65-401b-b77a-e2738013102f None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:07.202873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83871e8d-8d65-401b-b77a-e2738013102f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:07.203273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83871e8d-8d65-401b-b77a-e2738013102f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:07.223822 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-83871e8d-8d65-401b-b77a-e2738013102f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:07.225201 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-83871e8d-8d65-401b-b77a-e2738013102f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:07.320546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 953/1906] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:07.331375 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725ffeb9-c2a2-41be-9bfd-dc56e768428d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:07.332779 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-725ffeb9-c2a2-41be-9bfd-dc56e768428d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:07.398058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725ffeb9-c2a2-41be-9bfd-dc56e768428d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:07.402150 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4e6f43-6467-4c55-a789-a209ce4a6f46 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:07.402778 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4e6f43-6467-4c55-a789-a209ce4a6f46 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:07.403106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4e6f43-6467-4c55-a789-a209ce4a6f46 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:07.403379 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4e6f43-6467-4c55-a789-a209ce4a6f46 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:07.413376 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4e6f43-6467-4c55-a789-a209ce4a6f46 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:07.414806 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4e6f43-6467-4c55-a789-a209ce4a6f46 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:07.426092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 954/1907] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:07.691210 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:08.206375 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:08.791503 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83871e8d-8d65-401b-b77a-e2738013102f None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:08.793337 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-83871e8d-8d65-401b-b77a-e2738013102f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:08.806462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83871e8d-8d65-401b-b77a-e2738013102f None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:08.818418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3679b73e-680d-4b09-9634-88a4bdb3684a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:08.819245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3679b73e-680d-4b09-9634-88a4bdb3684a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:08.819599 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3679b73e-680d-4b09-9634-88a4bdb3684a admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:08.819956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3679b73e-680d-4b09-9634-88a4bdb3684a admin admin] PATH_INFO: `/v3/users/e0e8baca193c4c89b1f231ba002a91ed` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:08.821239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3679b73e-680d-4b09-9634-88a4bdb3684a admin admin] RBAC: Authorizing `identity:delete_user(user_id=e0e8baca193c4c89b1f231ba002a91ed)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:08.823511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3679b73e-680d-4b09-9634-88a4bdb3684a admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:09.033453 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-3679b73e-680d-4b09-9634-88a4bdb3684a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e0e8baca193c4c89b1f231ba002a91ed'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:09.153827 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-3679b73e-680d-4b09-9634-88a4bdb3684a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e0e8baca193c4c89b1f231ba002a91ed'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:09.158687 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8f7b48-8b0f-42d6-acad-07d2fbe756fb None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:09.159347 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8f7b48-8b0f-42d6-acad-07d2fbe756fb None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:09.159907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8f7b48-8b0f-42d6-acad-07d2fbe756fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:09.188247 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-3679b73e-680d-4b09-9634-88a4bdb3684a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e0e8baca193c4c89b1f231ba002a91ed'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:09.191940 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-3679b73e-680d-4b09-9634-88a4bdb3684a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e0e8baca193c4c89b1f231ba002a91ed'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:09.213321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-3679b73e-680d-4b09-9634-88a4bdb3684a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e0e8baca193c4c89b1f231ba002a91ed'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:09.236339 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2c8f7b48-8b0f-42d6-acad-07d2fbe756fb None None] MFA Rules not processed for user `bc5eb4c34a614860921f281fc7d4413c`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:09.241742 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 954/1908] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:49:08 2022] DELETE /identity/v3/users/e0e8baca193c4c89b1f231ba002a91ed => generated 0 bytes in 448 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:09.243331 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2875918-78d1-4fb1-8a68-ee31509f8934 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:09.244041 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2875918-78d1-4fb1-8a68-ee31509f8934 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:09.244432 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2875918-78d1-4fb1-8a68-ee31509f8934 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:09.245507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2c8f7b48-8b0f-42d6-acad-07d2fbe756fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:09.271990 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-a2875918-78d1-4fb1-8a68-ee31509f8934 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:09.273461 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a2875918-78d1-4fb1-8a68-ee31509f8934 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:09.432349 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 955/1909] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:09.433215 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2875918-78d1-4fb1-8a68-ee31509f8934 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:09.433708 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 955/1910] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:09 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:09.445303 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a2875918-78d1-4fb1-8a68-ee31509f8934 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:09.450569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8f7b48-8b0f-42d6-acad-07d2fbe756fb None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:09.452897 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2c8f7b48-8b0f-42d6-acad-07d2fbe756fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:09.471397 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2875918-78d1-4fb1-8a68-ee31509f8934 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:09.474114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883a15dc-f415-42e8-bf21-7da4eeb5c6b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:09.474114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883a15dc-f415-42e8-bf21-7da4eeb5c6b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:09.475252 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883a15dc-f415-42e8-bf21-7da4eeb5c6b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:09.475706 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883a15dc-f415-42e8-bf21-7da4eeb5c6b5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:09.476383 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883a15dc-f415-42e8-bf21-7da4eeb5c6b5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:09.478349 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883a15dc-f415-42e8-bf21-7da4eeb5c6b5 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:09.545828 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8f7b48-8b0f-42d6-acad-07d2fbe756fb None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:09.565593 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa5f31f-cc23-4d1a-99e6-21735f388cb5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:09.566288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa5f31f-cc23-4d1a-99e6-21735f388cb5 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:09.576598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa5f31f-cc23-4d1a-99e6-21735f388cb5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:09.576598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa5f31f-cc23-4d1a-99e6-21735f388cb5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:09.589334 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa5f31f-cc23-4d1a-99e6-21735f388cb5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:09.593008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa5f31f-cc23-4d1a-99e6-21735f388cb5 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:09.603639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 956/1911] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:49:09 2022] POST /identity/v3/projects => generated 373 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:09.607928 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883a15dc-f415-42e8-bf21-7da4eeb5c6b5 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:09.609796 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 956/1912] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:09.680270 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883a15dc-f415-42e8-bf21-7da4eeb5c6b5 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:09.688566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f2faac-776a-4dc4-9cb2-1726edeabf92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:09.689371 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f2faac-776a-4dc4-9cb2-1726edeabf92 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:09.689658 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f2faac-776a-4dc4-9cb2-1726edeabf92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:09.689937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f2faac-776a-4dc4-9cb2-1726edeabf92 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:09.690580 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f2faac-776a-4dc4-9cb2-1726edeabf92 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:09.692747 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f2faac-776a-4dc4-9cb2-1726edeabf92 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:09.817281 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 957/1913] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:49:09 2022] POST /identity/v3/users => generated 293 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:49:09.823528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa5f31f-cc23-4d1a-99e6-21735f388cb5 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:09.833857 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa5f31f-cc23-4d1a-99e6-21735f388cb5 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:09.838635 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65011960-f8f8-4b11-9c3c-e476339fbfd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:09.839261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65011960-f8f8-4b11-9c3c-e476339fbfd2 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:09.839543 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65011960-f8f8-4b11-9c3c-e476339fbfd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:09.839819 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65011960-f8f8-4b11-9c3c-e476339fbfd2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:09.840452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65011960-f8f8-4b11-9c3c-e476339fbfd2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:09.842540 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65011960-f8f8-4b11-9c3c-e476339fbfd2 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:09.852392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 957/1914] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:09 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:09.856809 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f2faac-776a-4dc4-9cb2-1726edeabf92 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:09.866989 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f2faac-776a-4dc4-9cb2-1726edeabf92 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:09.871634 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a6f9b7-d9c3-4c44-a073-7f92d97d1ff1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:09.872292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a6f9b7-d9c3-4c44-a073-7f92d97d1ff1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:09.873105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a6f9b7-d9c3-4c44-a073-7f92d97d1ff1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:09.873381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a6f9b7-d9c3-4c44-a073-7f92d97d1ff1 admin admin] PATH_INFO: `/v3/projects/f3ceb13de3b94aec80ec802c6a5061f1/users/342b2b7e7d0e4262b344aa00c57c11cc/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:09.891681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a6f9b7-d9c3-4c44-a073-7f92d97d1ff1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f3ceb13de3b94aec80ec802c6a5061f1, user_id=342b2b7e7d0e4262b344aa00c57c11cc, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:09.894704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a6f9b7-d9c3-4c44-a073-7f92d97d1ff1 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:09.984832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 958/1915] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:09 2022] PUT /identity/v3/projects/f3ceb13de3b94aec80ec802c6a5061f1/users/342b2b7e7d0e4262b344aa00c57c11cc/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:09.989710 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65011960-f8f8-4b11-9c3c-e476339fbfd2 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:10.004320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65011960-f8f8-4b11-9c3c-e476339fbfd2 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:10.075334 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b83a5b-f400-4774-b3c9-e4bf036d08c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:10.077028 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b83a5b-f400-4774-b3c9-e4bf036d08c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:10.077313 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b83a5b-f400-4774-b3c9-e4bf036d08c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:10.077645 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b83a5b-f400-4774-b3c9-e4bf036d08c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:10.078346 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b83a5b-f400-4774-b3c9-e4bf036d08c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:10.079960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b83a5b-f400-4774-b3c9-e4bf036d08c7 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:10.091326 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 958/1916] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:09 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:49:10.095716 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a6f9b7-d9c3-4c44-a073-7f92d97d1ff1 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:10.107763 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a6f9b7-d9c3-4c44-a073-7f92d97d1ff1 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:10.113526 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e2ead3-45d5-478b-ba17-f88ebac7c057 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:10.114264 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e2ead3-45d5-478b-ba17-f88ebac7c057 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:10.114542 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e2ead3-45d5-478b-ba17-f88ebac7c057 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:10.114862 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e2ead3-45d5-478b-ba17-f88ebac7c057 admin admin] PATH_INFO: `/v3/projects/f3ceb13de3b94aec80ec802c6a5061f1/users/342b2b7e7d0e4262b344aa00c57c11cc/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:10.116987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e2ead3-45d5-478b-ba17-f88ebac7c057 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f3ceb13de3b94aec80ec802c6a5061f1, user_id=342b2b7e7d0e4262b344aa00c57c11cc, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:10.119686 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e2ead3-45d5-478b-ba17-f88ebac7c057 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:10.131812 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 959/1917] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:10 2022] PUT /identity/v3/projects/f3ceb13de3b94aec80ec802c6a5061f1/users/342b2b7e7d0e4262b344aa00c57c11cc/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:10.155412 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b83a5b-f400-4774-b3c9-e4bf036d08c7 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:10.157284 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-73b83a5b-f400-4774-b3c9-e4bf036d08c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:10.179315 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b83a5b-f400-4774-b3c9-e4bf036d08c7 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:10.250376 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b85706-020a-4bd2-b886-d9fbef7674cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:10.251080 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b85706-020a-4bd2-b886-d9fbef7674cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:10.251353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b85706-020a-4bd2-b886-d9fbef7674cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:10.251631 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b85706-020a-4bd2-b886-d9fbef7674cd admin admin] PATH_INFO: `/v3/projects/1bd01cf0fc12498e9b594dd83c23a646` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:10.252765 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b85706-020a-4bd2-b886-d9fbef7674cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=1bd01cf0fc12498e9b594dd83c23a646)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:10.254958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b85706-020a-4bd2-b886-d9fbef7674cd admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:10.372477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:10.422962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b8b85706-020a-4bd2-b886-d9fbef7674cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1bd01cf0fc12498e9b594dd83c23a646', 'request_id': 'req-b8b85706-020a-4bd2-b886-d9fbef7674cd'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:10.430739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b8b85706-020a-4bd2-b886-d9fbef7674cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1bd01cf0fc12498e9b594dd83c23a646', 'request_id': 'req-b8b85706-020a-4bd2-b886-d9fbef7674cd'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:10.519650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b8b85706-020a-4bd2-b886-d9fbef7674cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1bd01cf0fc12498e9b594dd83c23a646', 'request_id': 'req-b8b85706-020a-4bd2-b886-d9fbef7674cd'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:10.533863 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b8b85706-020a-4bd2-b886-d9fbef7674cd admin admin] The token cache is being invalidate because project 1bd01cf0fc12498e9b594dd83c23a646 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:49:10.534196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b8b85706-020a-4bd2-b886-d9fbef7674cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:10.534883 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b8b85706-020a-4bd2-b886-d9fbef7674cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:10.537544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 959/1918] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:49:10 2022] DELETE /identity/v3/projects/1bd01cf0fc12498e9b594dd83c23a646 => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:10.541265 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e2ead3-45d5-478b-ba17-f88ebac7c057 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:10.543645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-90e2ead3-45d5-478b-ba17-f88ebac7c057 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:10.558074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e2ead3-45d5-478b-ba17-f88ebac7c057 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:10.636557 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32466387-aadd-410c-8e7a-a025a29f04c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:10.637135 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32466387-aadd-410c-8e7a-a025a29f04c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:10.637565 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32466387-aadd-410c-8e7a-a025a29f04c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:10.637854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32466387-aadd-410c-8e7a-a025a29f04c7 admin admin] PATH_INFO: `/v3/users/80c28c89b7734c74ae32dcf41e88f432` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:10.639233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32466387-aadd-410c-8e7a-a025a29f04c7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=80c28c89b7734c74ae32dcf41e88f432)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:10.641519 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32466387-aadd-410c-8e7a-a025a29f04c7 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:10.863812 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-32466387-aadd-410c-8e7a-a025a29f04c7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80c28c89b7734c74ae32dcf41e88f432'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:10.884508 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:10.980759 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-32466387-aadd-410c-8e7a-a025a29f04c7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80c28c89b7734c74ae32dcf41e88f432'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:10.995483 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-32466387-aadd-410c-8e7a-a025a29f04c7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80c28c89b7734c74ae32dcf41e88f432'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:10.996210 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-32466387-aadd-410c-8e7a-a025a29f04c7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80c28c89b7734c74ae32dcf41e88f432'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:11.004269 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-32466387-aadd-410c-8e7a-a025a29f04c7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80c28c89b7734c74ae32dcf41e88f432'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:11.011396 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 960/1919] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:49:10 2022] DELETE /identity/v3/users/80c28c89b7734c74ae32dcf41e88f432 => generated 0 bytes in 471 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:11.687214 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b85706-020a-4bd2-b886-d9fbef7674cd admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:11.703451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b85706-020a-4bd2-b886-d9fbef7674cd admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:11.777254 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb30dcd7-c52e-4393-bdb5-288a3381ec16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:11.777914 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb30dcd7-c52e-4393-bdb5-288a3381ec16 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:11.778339 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb30dcd7-c52e-4393-bdb5-288a3381ec16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:11.778617 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb30dcd7-c52e-4393-bdb5-288a3381ec16 admin admin] PATH_INFO: `/v3/projects/943f0e1e164148bab3612e84cf2891a6` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:11.780592 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb30dcd7-c52e-4393-bdb5-288a3381ec16 admin admin] RBAC: Authorizing `identity:delete_project(project_id=943f0e1e164148bab3612e84cf2891a6)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:11.782511 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb30dcd7-c52e-4393-bdb5-288a3381ec16 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:11.931602 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eb30dcd7-c52e-4393-bdb5-288a3381ec16 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '943f0e1e164148bab3612e84cf2891a6', 'request_id': 'req-eb30dcd7-c52e-4393-bdb5-288a3381ec16'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:11.945968 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eb30dcd7-c52e-4393-bdb5-288a3381ec16 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '943f0e1e164148bab3612e84cf2891a6', 'request_id': 'req-eb30dcd7-c52e-4393-bdb5-288a3381ec16'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:11.993789 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:12.033170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eb30dcd7-c52e-4393-bdb5-288a3381ec16 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '943f0e1e164148bab3612e84cf2891a6', 'request_id': 'req-eb30dcd7-c52e-4393-bdb5-288a3381ec16'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:12.050712 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eb30dcd7-c52e-4393-bdb5-288a3381ec16 admin admin] The token cache is being invalidate because project 943f0e1e164148bab3612e84cf2891a6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:49:12.051799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eb30dcd7-c52e-4393-bdb5-288a3381ec16 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:12.052872 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eb30dcd7-c52e-4393-bdb5-288a3381ec16 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:12.059349 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 960/1920] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:49:11 2022] DELETE /identity/v3/projects/943f0e1e164148bab3612e84cf2891a6 => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:12.063434 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d197648-f9c2-4fee-802a-eed08c353985 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:12.063744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d197648-f9c2-4fee-802a-eed08c353985 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:12.064017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d197648-f9c2-4fee-802a-eed08c353985 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:12.089995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-2d197648-f9c2-4fee-802a-eed08c353985 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:12.091296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-2d197648-f9c2-4fee-802a-eed08c353985 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:12.171071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 961/1921] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:12.173347 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4362690a-032f-4272-aadf-f376f9d519f7 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:12.173651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4362690a-032f-4272-aadf-f376f9d519f7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:12.173960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4362690a-032f-4272-aadf-f376f9d519f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:12.198112 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-4362690a-032f-4272-aadf-f376f9d519f7 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:12.199358 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4362690a-032f-4272-aadf-f376f9d519f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:12.207984 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 961/1922] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:12.212118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d197648-f9c2-4fee-802a-eed08c353985 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:12.222426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d197648-f9c2-4fee-802a-eed08c353985 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:12.227420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77bb83b-54d5-4949-ac6f-b73f71f6e0ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:12.228053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77bb83b-54d5-4949-ac6f-b73f71f6e0ea admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:12.228544 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77bb83b-54d5-4949-ac6f-b73f71f6e0ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:12.228720 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77bb83b-54d5-4949-ac6f-b73f71f6e0ea admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:12.229438 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77bb83b-54d5-4949-ac6f-b73f71f6e0ea admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:49:12.229814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77bb83b-54d5-4949-ac6f-b73f71f6e0ea admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:12.231458 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77bb83b-54d5-4949-ac6f-b73f71f6e0ea admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:12.253045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 962/1923] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:49:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:49:12.253145 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4362690a-032f-4272-aadf-f376f9d519f7 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:12.261390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4362690a-032f-4272-aadf-f376f9d519f7 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:12.267188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d71bb56-4fa5-49a2-a6ae-f23c24dac823 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:12.267466 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d71bb56-4fa5-49a2-a6ae-f23c24dac823 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:12.267745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d71bb56-4fa5-49a2-a6ae-f23c24dac823 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:12.268014 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d71bb56-4fa5-49a2-a6ae-f23c24dac823 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:12.268653 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d71bb56-4fa5-49a2-a6ae-f23c24dac823 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:12.271383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d71bb56-4fa5-49a2-a6ae-f23c24dac823 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:12.376077 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 962/1924] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:49:12 2022] POST /identity/v3/projects => generated 381 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:49:12.377309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77bb83b-54d5-4949-ac6f-b73f71f6e0ea admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:12.389228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77bb83b-54d5-4949-ac6f-b73f71f6e0ea admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:12.476335 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263e09cb-0430-4bf3-903b-80359541eaa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:12.476967 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263e09cb-0430-4bf3-903b-80359541eaa0 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:12.477249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263e09cb-0430-4bf3-903b-80359541eaa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:12.477527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263e09cb-0430-4bf3-903b-80359541eaa0 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:12.478295 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263e09cb-0430-4bf3-903b-80359541eaa0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:12.480438 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263e09cb-0430-4bf3-903b-80359541eaa0 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:12.588752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 963/1925] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:49:12 2022] POST /identity/v3/users => generated 291 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:12.592868 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d71bb56-4fa5-49a2-a6ae-f23c24dac823 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:12.606311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d71bb56-4fa5-49a2-a6ae-f23c24dac823 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:12.611591 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1349da-7615-4d58-b347-3e8568ffa74a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:12.612303 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1349da-7615-4d58-b347-3e8568ffa74a admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:12.612671 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1349da-7615-4d58-b347-3e8568ffa74a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:12.613023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1349da-7615-4d58-b347-3e8568ffa74a admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:12.613741 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1349da-7615-4d58-b347-3e8568ffa74a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:12.615740 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1349da-7615-4d58-b347-3e8568ffa74a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:12.630655 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 963/1926] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:12 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:12.634604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263e09cb-0430-4bf3-903b-80359541eaa0 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:12.652111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263e09cb-0430-4bf3-903b-80359541eaa0 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:12.657917 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02aee8d-b4c7-4157-94af-cf425ce36591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:12.658551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02aee8d-b4c7-4157-94af-cf425ce36591 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:12.658812 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02aee8d-b4c7-4157-94af-cf425ce36591 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:12.659156 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02aee8d-b4c7-4157-94af-cf425ce36591 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:12.659783 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02aee8d-b4c7-4157-94af-cf425ce36591 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:12.661509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02aee8d-b4c7-4157-94af-cf425ce36591 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:12.672767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 964/1927] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:12 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:12.677385 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1349da-7615-4d58-b347-3e8568ffa74a admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:12.687888 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1349da-7615-4d58-b347-3e8568ffa74a admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:12.692797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4332c4-8e64-4bb9-a3f5-ff0f2160a828 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:12.693447 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4332c4-8e64-4bb9-a3f5-ff0f2160a828 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:12.693734 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4332c4-8e64-4bb9-a3f5-ff0f2160a828 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:12.694014 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4332c4-8e64-4bb9-a3f5-ff0f2160a828 admin admin] PATH_INFO: `/v3/projects/d614450b1c424815b09fdce5c5ae389b/users/bcbc96070b1f4c4eb23536c5c2f2a7a9/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:12.709311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4332c4-8e64-4bb9-a3f5-ff0f2160a828 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d614450b1c424815b09fdce5c5ae389b, user_id=bcbc96070b1f4c4eb23536c5c2f2a7a9, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:12.712101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4332c4-8e64-4bb9-a3f5-ff0f2160a828 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:12.822535 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 964/1928] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:12 2022] PUT /identity/v3/projects/d614450b1c424815b09fdce5c5ae389b/users/bcbc96070b1f4c4eb23536c5c2f2a7a9/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:12.833994 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f734d5-3080-4405-82d7-0662841d73f5 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:12.834305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f734d5-3080-4405-82d7-0662841d73f5 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:12.834617 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f734d5-3080-4405-82d7-0662841d73f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:12.863765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-54f734d5-3080-4405-82d7-0662841d73f5 None None] MFA Rules not processed for user `bcbc96070b1f4c4eb23536c5c2f2a7a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:12.864977 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-54f734d5-3080-4405-82d7-0662841d73f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:12.948568 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 965/1929] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:12 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:12.949644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:12.952800 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4332c4-8e64-4bb9-a3f5-ff0f2160a828 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:13.010966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fee4ba-2556-40aa-b6ec-c99f6663124d None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:13.011496 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fee4ba-2556-40aa-b6ec-c99f6663124d None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:13.011681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fee4ba-2556-40aa-b6ec-c99f6663124d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:13.067529 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-d1fee4ba-2556-40aa-b6ec-c99f6663124d None None] MFA Rules not processed for user `342b2b7e7d0e4262b344aa00c57c11cc`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:13.071207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d1fee4ba-2556-40aa-b6ec-c99f6663124d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:13.072245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4332c4-8e64-4bb9-a3f5-ff0f2160a828 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:13.083172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bd00bc-6ac9-4c8d-a9de-696990630c13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:13.083955 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bd00bc-6ac9-4c8d-a9de-696990630c13 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:13.084246 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bd00bc-6ac9-4c8d-a9de-696990630c13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:13.085532 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bd00bc-6ac9-4c8d-a9de-696990630c13 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:13.086211 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58bd00bc-6ac9-4c8d-a9de-696990630c13 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:13.088150 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58bd00bc-6ac9-4c8d-a9de-696990630c13 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:13.196268 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 966/1930] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:13 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:13.210488 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fee4ba-2556-40aa-b6ec-c99f6663124d None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:13.216228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d1fee4ba-2556-40aa-b6ec-c99f6663124d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:13.306965 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 965/1931] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:49:12 2022] POST /identity/v3/projects => generated 381 bytes in 353 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:13.309531 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bd00bc-6ac9-4c8d-a9de-696990630c13 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:13.339281 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fee4ba-2556-40aa-b6ec-c99f6663124d None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:13.390445 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bd00bc-6ac9-4c8d-a9de-696990630c13 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:13.395805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b098ea-4017-4914-8c7e-3a2c6a2fcfaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:13.396589 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b098ea-4017-4914-8c7e-3a2c6a2fcfaf admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:13.396870 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b098ea-4017-4914-8c7e-3a2c6a2fcfaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:13.397147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b098ea-4017-4914-8c7e-3a2c6a2fcfaf admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:13.397790 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b098ea-4017-4914-8c7e-3a2c6a2fcfaf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:13.399862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b098ea-4017-4914-8c7e-3a2c6a2fcfaf admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:13.417177 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3893b451-5e16-4e5b-8f38-36939995a331 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:13.418528 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3893b451-5e16-4e5b-8f38-36939995a331 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:13.418528 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3893b451-5e16-4e5b-8f38-36939995a331 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:13.419048 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3893b451-5e16-4e5b-8f38-36939995a331 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:13.421701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3893b451-5e16-4e5b-8f38-36939995a331 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:13.502750 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 966/1932] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:49:13 2022] POST /identity/v3/users => generated 297 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:13.507130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b098ea-4017-4914-8c7e-3a2c6a2fcfaf admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:13.514320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3893b451-5e16-4e5b-8f38-36939995a331 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:13.515820 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3893b451-5e16-4e5b-8f38-36939995a331 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:13.518732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b098ea-4017-4914-8c7e-3a2c6a2fcfaf admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:13.528179 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced6614c-9caa-4cf9-9305-bc434adbfdbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:13.537660 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 967/1933] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:13 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:13.539349 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced6614c-9caa-4cf9-9305-bc434adbfdbe admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:13.539847 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced6614c-9caa-4cf9-9305-bc434adbfdbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:13.540227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced6614c-9caa-4cf9-9305-bc434adbfdbe admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:13.541229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced6614c-9caa-4cf9-9305-bc434adbfdbe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:13.543766 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced6614c-9caa-4cf9-9305-bc434adbfdbe admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:13.568230 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 967/1934] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:13 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:13.568351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3893b451-5e16-4e5b-8f38-36939995a331 service cinder] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:13.583183 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3893b451-5e16-4e5b-8f38-36939995a331 service cinder] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:13.589371 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980f12c6-b55c-4d6e-bf60-d1eccb89693c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:13.590143 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980f12c6-b55c-4d6e-bf60-d1eccb89693c admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:13.590493 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980f12c6-b55c-4d6e-bf60-d1eccb89693c admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:13.590836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980f12c6-b55c-4d6e-bf60-d1eccb89693c admin admin] PATH_INFO: `/v3/projects/5f959c20d356474fb7249e321965d4de/users/d9749331c0d043acbecffcb3d9d6ecc8/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:13.611933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980f12c6-b55c-4d6e-bf60-d1eccb89693c admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f959c20d356474fb7249e321965d4de, user_id=d9749331c0d043acbecffcb3d9d6ecc8, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:13.616551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980f12c6-b55c-4d6e-bf60-d1eccb89693c admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:13.734312 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 968/1935] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:13 2022] PUT /identity/v3/projects/5f959c20d356474fb7249e321965d4de/users/d9749331c0d043acbecffcb3d9d6ecc8/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:49:13.752829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced6614c-9caa-4cf9-9305-bc434adbfdbe admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:13.769502 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced6614c-9caa-4cf9-9305-bc434adbfdbe admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:13.845291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=93,keystone:INSERT=2,keystone:DELETE=5 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:13.910495 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b727c577-1b5e-4ff1-99ee-38584aedbc18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:13.911280 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b727c577-1b5e-4ff1-99ee-38584aedbc18 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:13.911705 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b727c577-1b5e-4ff1-99ee-38584aedbc18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:13.912105 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b727c577-1b5e-4ff1-99ee-38584aedbc18 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:13.913843 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b727c577-1b5e-4ff1-99ee-38584aedbc18 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:13.915918 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b727c577-1b5e-4ff1-99ee-38584aedbc18 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:13.945991 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 968/1936] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:13 2022] GET /identity/v3/roles => generated 1420 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:13.950339 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980f12c6-b55c-4d6e-bf60-d1eccb89693c admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:13.994060 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980f12c6-b55c-4d6e-bf60-d1eccb89693c admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:14.011720 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2d15d5-b8fd-4d76-9e7e-13f0b7d60bdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:14.014108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2d15d5-b8fd-4d76-9e7e-13f0b7d60bdd admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:14.014491 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2d15d5-b8fd-4d76-9e7e-13f0b7d60bdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:14.014774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2d15d5-b8fd-4d76-9e7e-13f0b7d60bdd admin admin] PATH_INFO: `/v3/projects/5f959c20d356474fb7249e321965d4de/users/d9749331c0d043acbecffcb3d9d6ecc8/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:14.021397 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2d15d5-b8fd-4d76-9e7e-13f0b7d60bdd admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f959c20d356474fb7249e321965d4de, user_id=d9749331c0d043acbecffcb3d9d6ecc8, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:14.024698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2d15d5-b8fd-4d76-9e7e-13f0b7d60bdd admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:14.041823 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 969/1937] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:13 2022] PUT /identity/v3/projects/5f959c20d356474fb7249e321965d4de/users/d9749331c0d043acbecffcb3d9d6ecc8/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:14.053425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e84b11f-21fb-45c3-8fad-619b4b1f5225 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:14.053716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e84b11f-21fb-45c3-8fad-619b4b1f5225 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:14.054026 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e84b11f-21fb-45c3-8fad-619b4b1f5225 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:14.102409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1e84b11f-21fb-45c3-8fad-619b4b1f5225 None None] MFA Rules not processed for user `d9749331c0d043acbecffcb3d9d6ecc8`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:14.104714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1e84b11f-21fb-45c3-8fad-619b4b1f5225 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:14.238828 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 969/1938] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:14 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:14.332344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:DELETE=20,keystone:SELECT=78,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:14.411680 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2d15d5-b8fd-4d76-9e7e-13f0b7d60bdd admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:14.503473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2d15d5-b8fd-4d76-9e7e-13f0b7d60bdd admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:14.510352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e989db1a-ddd4-47db-891b-d4038285617b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:14.510935 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e989db1a-ddd4-47db-891b-d4038285617b service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:14.511233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e989db1a-ddd4-47db-891b-d4038285617b service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:14.512274 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e989db1a-ddd4-47db-891b-d4038285617b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:14.537296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e989db1a-ddd4-47db-891b-d4038285617b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:14.539195 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e989db1a-ddd4-47db-891b-d4038285617b service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:14.552140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 970/1939] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:14 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:15.398072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e84b11f-21fb-45c3-8fad-619b4b1f5225 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:15.487104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e84b11f-21fb-45c3-8fad-619b4b1f5225 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:15.491916 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c78a32-45e0-4c1f-8943-a9f56e54ba36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:15.492706 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c78a32-45e0-4c1f-8943-a9f56e54ba36 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:15.493043 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c78a32-45e0-4c1f-8943-a9f56e54ba36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:15.493330 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c78a32-45e0-4c1f-8943-a9f56e54ba36 admin admin] PATH_INFO: `/v3/users/bcbc96070b1f4c4eb23536c5c2f2a7a9` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:15.495644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c78a32-45e0-4c1f-8943-a9f56e54ba36 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bcbc96070b1f4c4eb23536c5c2f2a7a9)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:15.496447 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c78a32-45e0-4c1f-8943-a9f56e54ba36 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:15.829091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e989db1a-ddd4-47db-891b-d4038285617b service cinder] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:15.830117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d1c78a32-45e0-4c1f-8943-a9f56e54ba36 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bcbc96070b1f4c4eb23536c5c2f2a7a9'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:15.832756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e989db1a-ddd4-47db-891b-d4038285617b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:15.855378 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e989db1a-ddd4-47db-891b-d4038285617b service cinder] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:15.861278 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d1c78a32-45e0-4c1f-8943-a9f56e54ba36 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bcbc96070b1f4c4eb23536c5c2f2a7a9'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:15.867148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cfa088-1207-4bf3-b671-f57ad89e823d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:15.867148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cfa088-1207-4bf3-b671-f57ad89e823d service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:15.867148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cfa088-1207-4bf3-b671-f57ad89e823d service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:15.867148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cfa088-1207-4bf3-b671-f57ad89e823d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:15.962236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d1c78a32-45e0-4c1f-8943-a9f56e54ba36 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bcbc96070b1f4c4eb23536c5c2f2a7a9'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:15.964756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d1c78a32-45e0-4c1f-8943-a9f56e54ba36 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bcbc96070b1f4c4eb23536c5c2f2a7a9'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:15.983383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d1c78a32-45e0-4c1f-8943-a9f56e54ba36 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bcbc96070b1f4c4eb23536c5c2f2a7a9'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:16.001276 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 970/1940] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:49:15 2022] DELETE /identity/v3/users/bcbc96070b1f4c4eb23536c5c2f2a7a9 => generated 0 bytes in 603 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:16.004827 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24cfa088-1207-4bf3-b671-f57ad89e823d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:16.006895 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24cfa088-1207-4bf3-b671-f57ad89e823d service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:16.012112 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bafcb5a-b632-41b2-8ee3-0db783a68a64 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:16.012588 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bafcb5a-b632-41b2-8ee3-0db783a68a64 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:16.014145 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bafcb5a-b632-41b2-8ee3-0db783a68a64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:16.019696 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 971/1941] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:15 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:16.062223 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9bafcb5a-b632-41b2-8ee3-0db783a68a64 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:16.063663 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9bafcb5a-b632-41b2-8ee3-0db783a68a64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:16.148519 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 971/1942] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:16.160278 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cfa088-1207-4bf3-b671-f57ad89e823d service cinder] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:16.162057 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-24cfa088-1207-4bf3-b671-f57ad89e823d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:16.332362 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cfa088-1207-4bf3-b671-f57ad89e823d service cinder] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:16.454865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5b7f98-4ace-4157-8db6-4662f2fa3760 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:16.455502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5b7f98-4ace-4157-8db6-4662f2fa3760 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:16.455803 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5b7f98-4ace-4157-8db6-4662f2fa3760 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:16.456051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5b7f98-4ace-4157-8db6-4662f2fa3760 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:16.473379 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5b7f98-4ace-4157-8db6-4662f2fa3760 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:16.474824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5b7f98-4ace-4157-8db6-4662f2fa3760 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:16.485393 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 972/1943] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:16.691316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:17.297604 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:17.316548 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bafcb5a-b632-41b2-8ee3-0db783a68a64 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:17.318578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9bafcb5a-b632-41b2-8ee3-0db783a68a64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:17.348010 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bafcb5a-b632-41b2-8ee3-0db783a68a64 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:17.348010 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74e9868-209c-4d26-abf8-d895e644d909 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:17.348010 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74e9868-209c-4d26-abf8-d895e644d909 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:17.348646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74e9868-209c-4d26-abf8-d895e644d909 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:17.348646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74e9868-209c-4d26-abf8-d895e644d909 admin admin] PATH_INFO: `/v3/projects/d614450b1c424815b09fdce5c5ae389b` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:17.349653 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74e9868-209c-4d26-abf8-d895e644d909 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d614450b1c424815b09fdce5c5ae389b)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:17.351638 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74e9868-209c-4d26-abf8-d895e644d909 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:17.521193 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c74e9868-209c-4d26-abf8-d895e644d909 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd614450b1c424815b09fdce5c5ae389b', 'request_id': 'req-c74e9868-209c-4d26-abf8-d895e644d909'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:17.541732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c74e9868-209c-4d26-abf8-d895e644d909 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd614450b1c424815b09fdce5c5ae389b', 'request_id': 'req-c74e9868-209c-4d26-abf8-d895e644d909'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:17.621803 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c74e9868-209c-4d26-abf8-d895e644d909 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd614450b1c424815b09fdce5c5ae389b', 'request_id': 'req-c74e9868-209c-4d26-abf8-d895e644d909'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:17.635421 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c74e9868-209c-4d26-abf8-d895e644d909 admin admin] The token cache is being invalidate because project d614450b1c424815b09fdce5c5ae389b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:49:17.635752 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c74e9868-209c-4d26-abf8-d895e644d909 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:17.636708 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c74e9868-209c-4d26-abf8-d895e644d909 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:17.641658 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 972/1944] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:49:17 2022] DELETE /identity/v3/projects/d614450b1c424815b09fdce5c5ae389b => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:17.642885 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5b7f98-4ace-4157-8db6-4662f2fa3760 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:17.654820 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5b7f98-4ace-4157-8db6-4662f2fa3760 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:17.717427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2e2ea8-f475-4198-9de3-8c06aa12080b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:17.718507 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2e2ea8-f475-4198-9de3-8c06aa12080b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:17.719103 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2e2ea8-f475-4198-9de3-8c06aa12080b admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:17.719531 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2e2ea8-f475-4198-9de3-8c06aa12080b admin admin] PATH_INFO: `/v3/users/d9749331c0d043acbecffcb3d9d6ecc8` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:17.721119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e2e2ea8-f475-4198-9de3-8c06aa12080b admin admin] RBAC: Authorizing `identity:delete_user(user_id=d9749331c0d043acbecffcb3d9d6ecc8)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:17.723188 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e2e2ea8-f475-4198-9de3-8c06aa12080b admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:17.963448 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-7e2e2ea8-f475-4198-9de3-8c06aa12080b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd9749331c0d043acbecffcb3d9d6ecc8'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:18.046368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-7e2e2ea8-f475-4198-9de3-8c06aa12080b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd9749331c0d043acbecffcb3d9d6ecc8'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:18.072882 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-7e2e2ea8-f475-4198-9de3-8c06aa12080b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd9749331c0d043acbecffcb3d9d6ecc8'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:18.073836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-7e2e2ea8-f475-4198-9de3-8c06aa12080b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd9749331c0d043acbecffcb3d9d6ecc8'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:18.083179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-7e2e2ea8-f475-4198-9de3-8c06aa12080b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd9749331c0d043acbecffcb3d9d6ecc8'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:18.089126 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 973/1945] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:49:17 2022] DELETE /identity/v3/users/d9749331c0d043acbecffcb3d9d6ecc8 => generated 0 bytes in 447 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:18.663817 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74e9868-209c-4d26-abf8-d895e644d909 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:18.665355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c74e9868-209c-4d26-abf8-d895e644d909 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:18.686234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74e9868-209c-4d26-abf8-d895e644d909 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:18.756748 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043ccd1b-51af-449c-9f1a-f28da2834016 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:18.757383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043ccd1b-51af-449c-9f1a-f28da2834016 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:18.757669 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043ccd1b-51af-449c-9f1a-f28da2834016 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:18.758006 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043ccd1b-51af-449c-9f1a-f28da2834016 admin admin] PATH_INFO: `/v3/projects/5f959c20d356474fb7249e321965d4de` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:18.759657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043ccd1b-51af-449c-9f1a-f28da2834016 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5f959c20d356474fb7249e321965d4de)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:18.761277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043ccd1b-51af-449c-9f1a-f28da2834016 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:18.881423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-043ccd1b-51af-449c-9f1a-f28da2834016 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5f959c20d356474fb7249e321965d4de', 'request_id': 'req-043ccd1b-51af-449c-9f1a-f28da2834016'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:18.889576 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-043ccd1b-51af-449c-9f1a-f28da2834016 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5f959c20d356474fb7249e321965d4de', 'request_id': 'req-043ccd1b-51af-449c-9f1a-f28da2834016'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:18.993305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-043ccd1b-51af-449c-9f1a-f28da2834016 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5f959c20d356474fb7249e321965d4de', 'request_id': 'req-043ccd1b-51af-449c-9f1a-f28da2834016'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:19.004828 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-043ccd1b-51af-449c-9f1a-f28da2834016 admin admin] The token cache is being invalidate because project 5f959c20d356474fb7249e321965d4de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:49:19.005117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-043ccd1b-51af-449c-9f1a-f28da2834016 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:19.005774 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-043ccd1b-51af-449c-9f1a-f28da2834016 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:19.007322 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 973/1946] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:49:18 2022] DELETE /identity/v3/projects/5f959c20d356474fb7249e321965d4de => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:19.012248 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949d1c14-6106-47dc-961f-daa2350aee7d None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:19.012693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949d1c14-6106-47dc-961f-daa2350aee7d None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:19.012782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949d1c14-6106-47dc-961f-daa2350aee7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:19.022416 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:19.034490 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-949d1c14-6106-47dc-961f-daa2350aee7d None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:19.035733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-949d1c14-6106-47dc-961f-daa2350aee7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:19.097141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 974/1947] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:19.104868 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb905f4-ca3a-41d2-bbc4-82022dd43832 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:19.105152 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb905f4-ca3a-41d2-bbc4-82022dd43832 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:19.105477 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb905f4-ca3a-41d2-bbc4-82022dd43832 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:19.126301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1eb905f4-ca3a-41d2-bbc4-82022dd43832 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:19.127544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1eb905f4-ca3a-41d2-bbc4-82022dd43832 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:19.135416 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 974/1948] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:19.140467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949d1c14-6106-47dc-961f-daa2350aee7d None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:19.150028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949d1c14-6106-47dc-961f-daa2350aee7d None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:19.155262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39070205-0eed-415c-92f9-055b9da489cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:19.155880 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39070205-0eed-415c-92f9-055b9da489cd admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:19.156157 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39070205-0eed-415c-92f9-055b9da489cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:19.156433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39070205-0eed-415c-92f9-055b9da489cd admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:19.157152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39070205-0eed-415c-92f9-055b9da489cd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:49:19.157384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39070205-0eed-415c-92f9-055b9da489cd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:19.160626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39070205-0eed-415c-92f9-055b9da489cd admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:19.179122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 975/1949] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:49:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:49:19.183175 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb905f4-ca3a-41d2-bbc4-82022dd43832 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:19.194431 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb905f4-ca3a-41d2-bbc4-82022dd43832 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:19.202541 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99db62a7-8c21-4a99-8c02-61338bf5dc28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:19.203263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99db62a7-8c21-4a99-8c02-61338bf5dc28 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:19.203490 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99db62a7-8c21-4a99-8c02-61338bf5dc28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:19.203804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99db62a7-8c21-4a99-8c02-61338bf5dc28 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:19.204459 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99db62a7-8c21-4a99-8c02-61338bf5dc28 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:19.206430 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99db62a7-8c21-4a99-8c02-61338bf5dc28 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:19.311399 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 975/1950] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:49:19 2022] POST /identity/v3/projects => generated 373 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:19.317738 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39070205-0eed-415c-92f9-055b9da489cd admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:19.334301 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39070205-0eed-415c-92f9-055b9da489cd admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:19.365622 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:19.399858 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf1f384-ada8-449f-9dda-4da72a5192d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:19.400472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf1f384-ada8-449f-9dda-4da72a5192d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:19.400789 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf1f384-ada8-449f-9dda-4da72a5192d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:19.401083 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf1f384-ada8-449f-9dda-4da72a5192d0 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:19.401757 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf1f384-ada8-449f-9dda-4da72a5192d0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:19.404385 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf1f384-ada8-449f-9dda-4da72a5192d0 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:19.497922 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 976/1951] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:49:19 2022] POST /identity/v3/users => generated 287 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:19.502521 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99db62a7-8c21-4a99-8c02-61338bf5dc28 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:19.513957 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99db62a7-8c21-4a99-8c02-61338bf5dc28 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:19.518725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5646bbf8-0dda-4451-a9d6-0465e98d6336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:19.519379 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5646bbf8-0dda-4451-a9d6-0465e98d6336 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:19.519666 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5646bbf8-0dda-4451-a9d6-0465e98d6336 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:19.519942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5646bbf8-0dda-4451-a9d6-0465e98d6336 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:19.520598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5646bbf8-0dda-4451-a9d6-0465e98d6336 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:19.522196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5646bbf8-0dda-4451-a9d6-0465e98d6336 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:19.533056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 976/1952] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:19 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:19.537225 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf1f384-ada8-449f-9dda-4da72a5192d0 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:19.547784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf1f384-ada8-449f-9dda-4da72a5192d0 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:19.552984 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92cf3ff-f102-4bed-bfec-f653cca27fbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:19.553620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92cf3ff-f102-4bed-bfec-f653cca27fbc admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:19.553903 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92cf3ff-f102-4bed-bfec-f653cca27fbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:19.554188 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92cf3ff-f102-4bed-bfec-f653cca27fbc admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:19.554940 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92cf3ff-f102-4bed-bfec-f653cca27fbc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:19.556676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92cf3ff-f102-4bed-bfec-f653cca27fbc admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:19.570144 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 977/1953] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:19 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:19.574527 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5646bbf8-0dda-4451-a9d6-0465e98d6336 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:19.585769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5646bbf8-0dda-4451-a9d6-0465e98d6336 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:19.591307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f6933b-53e0-48d1-a610-8d6ef21020c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:19.591833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f6933b-53e0-48d1-a610-8d6ef21020c9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:19.592524 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f6933b-53e0-48d1-a610-8d6ef21020c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:19.592986 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f6933b-53e0-48d1-a610-8d6ef21020c9 admin admin] PATH_INFO: `/v3/projects/5c817d8a98034c56a40a68e87dd10ca0/users/dd3b57590fb94dcb86351df9ea596d30/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:19.610522 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f6933b-53e0-48d1-a610-8d6ef21020c9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c817d8a98034c56a40a68e87dd10ca0, user_id=dd3b57590fb94dcb86351df9ea596d30, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:19.613233 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f6933b-53e0-48d1-a610-8d6ef21020c9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:19.680246 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 977/1954] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:19 2022] PUT /identity/v3/projects/5c817d8a98034c56a40a68e87dd10ca0/users/dd3b57590fb94dcb86351df9ea596d30/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:19.691883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7fe568-b44d-473f-bfe7-dfcf3ad77940 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:19.692180 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7fe568-b44d-473f-bfe7-dfcf3ad77940 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:19.692449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7fe568-b44d-473f-bfe7-dfcf3ad77940 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:19.722584 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-ec7fe568-b44d-473f-bfe7-dfcf3ad77940 None None] MFA Rules not processed for user `dd3b57590fb94dcb86351df9ea596d30`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:19.723876 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ec7fe568-b44d-473f-bfe7-dfcf3ad77940 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:19.789134 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 978/1955] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:19 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:19.800530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f6933b-53e0-48d1-a610-8d6ef21020c9 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:19.804026 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-20f6933b-53e0-48d1-a610-8d6ef21020c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:19.865508 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f6933b-53e0-48d1-a610-8d6ef21020c9 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:19.869441 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf43be1-0d91-468b-a24c-0f5b7f40e2b2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:19.870062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf43be1-0d91-468b-a24c-0f5b7f40e2b2 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:19.870346 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf43be1-0d91-468b-a24c-0f5b7f40e2b2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:19.870644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf43be1-0d91-468b-a24c-0f5b7f40e2b2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:19.880138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf43be1-0d91-468b-a24c-0f5b7f40e2b2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:19.881543 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf43be1-0d91-468b-a24c-0f5b7f40e2b2 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:19.891639 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 978/1956] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:19 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:19.914940 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7fe568-b44d-473f-bfe7-dfcf3ad77940 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:19.978211 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7fe568-b44d-473f-bfe7-dfcf3ad77940 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:19.982976 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a69989a-014b-4204-8463-c61a50d803f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:19.983609 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a69989a-014b-4204-8463-c61a50d803f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:19.984234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a69989a-014b-4204-8463-c61a50d803f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:19.984234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a69989a-014b-4204-8463-c61a50d803f0 admin admin] PATH_INFO: `/v3/users/dd3b57590fb94dcb86351df9ea596d30` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:19.985278 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a69989a-014b-4204-8463-c61a50d803f0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dd3b57590fb94dcb86351df9ea596d30)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:19.987321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a69989a-014b-4204-8463-c61a50d803f0 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:20.247282 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5a69989a-014b-4204-8463-c61a50d803f0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dd3b57590fb94dcb86351df9ea596d30'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:20.319017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5a69989a-014b-4204-8463-c61a50d803f0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dd3b57590fb94dcb86351df9ea596d30'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:20.332921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5a69989a-014b-4204-8463-c61a50d803f0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dd3b57590fb94dcb86351df9ea596d30'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:20.333621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5a69989a-014b-4204-8463-c61a50d803f0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dd3b57590fb94dcb86351df9ea596d30'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:20.340605 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5a69989a-014b-4204-8463-c61a50d803f0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dd3b57590fb94dcb86351df9ea596d30'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:20.346894 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 979/1957] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:49:19 2022] DELETE /identity/v3/users/dd3b57590fb94dcb86351df9ea596d30 => generated 0 bytes in 433 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:20.351371 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80133a2d-df03-48cf-8ff5-f7f667a4d7fc None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:20.351652 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80133a2d-df03-48cf-8ff5-f7f667a4d7fc None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:20.351931 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80133a2d-df03-48cf-8ff5-f7f667a4d7fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:20.370690 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-80133a2d-df03-48cf-8ff5-f7f667a4d7fc None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:20.371936 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-80133a2d-df03-48cf-8ff5-f7f667a4d7fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:20.432343 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 979/1958] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:20.442716 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a69989a-014b-4204-8463-c61a50d803f0 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:20.444124 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-5a69989a-014b-4204-8463-c61a50d803f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:20.500052 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a69989a-014b-4204-8463-c61a50d803f0 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:20.503630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e7aed9-e516-4e1a-8645-dd2ae140d9ba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:20.504235 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e7aed9-e516-4e1a-8645-dd2ae140d9ba service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:20.504514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e7aed9-e516-4e1a-8645-dd2ae140d9ba service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:20.504783 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e7aed9-e516-4e1a-8645-dd2ae140d9ba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:20.518554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e7aed9-e516-4e1a-8645-dd2ae140d9ba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:20.518554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e7aed9-e516-4e1a-8645-dd2ae140d9ba service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:20.525457 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 980/1959] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:20.542115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:20.782397 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:21.045827 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80133a2d-df03-48cf-8ff5-f7f667a4d7fc None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:21.047563 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-80133a2d-df03-48cf-8ff5-f7f667a4d7fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:21.063495 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80133a2d-df03-48cf-8ff5-f7f667a4d7fc None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:21.069118 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018fc0ff-33c2-4736-a208-5ae9c7caebd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:21.070067 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018fc0ff-33c2-4736-a208-5ae9c7caebd9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:21.070488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018fc0ff-33c2-4736-a208-5ae9c7caebd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:21.070979 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018fc0ff-33c2-4736-a208-5ae9c7caebd9 admin admin] PATH_INFO: `/v3/projects/5c817d8a98034c56a40a68e87dd10ca0` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:21.072547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018fc0ff-33c2-4736-a208-5ae9c7caebd9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c817d8a98034c56a40a68e87dd10ca0)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:21.074637 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018fc0ff-33c2-4736-a208-5ae9c7caebd9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:21.236208 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-018fc0ff-33c2-4736-a208-5ae9c7caebd9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c817d8a98034c56a40a68e87dd10ca0', 'request_id': 'req-018fc0ff-33c2-4736-a208-5ae9c7caebd9'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:21.243077 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-018fc0ff-33c2-4736-a208-5ae9c7caebd9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c817d8a98034c56a40a68e87dd10ca0', 'request_id': 'req-018fc0ff-33c2-4736-a208-5ae9c7caebd9'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:21.327616 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-018fc0ff-33c2-4736-a208-5ae9c7caebd9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c817d8a98034c56a40a68e87dd10ca0', 'request_id': 'req-018fc0ff-33c2-4736-a208-5ae9c7caebd9'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:21.339992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-018fc0ff-33c2-4736-a208-5ae9c7caebd9 admin admin] The token cache is being invalidate because project 5c817d8a98034c56a40a68e87dd10ca0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:49:21.340412 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-018fc0ff-33c2-4736-a208-5ae9c7caebd9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:21.341182 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-018fc0ff-33c2-4736-a208-5ae9c7caebd9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:21.343977 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 980/1960] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:49:21 2022] DELETE /identity/v3/projects/5c817d8a98034c56a40a68e87dd10ca0 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:49:21.350111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15156d7-5681-41ca-b7fb-579ae5ea5260 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:21.350987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15156d7-5681-41ca-b7fb-579ae5ea5260 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:21.351273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15156d7-5681-41ca-b7fb-579ae5ea5260 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:21.372217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-a15156d7-5681-41ca-b7fb-579ae5ea5260 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:21.373613 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a15156d7-5681-41ca-b7fb-579ae5ea5260 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:21.441286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 981/1961] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:21.450408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe05855a-7e8e-49cb-95f1-f57089c33150 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:21.450770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe05855a-7e8e-49cb-95f1-f57089c33150 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:21.451065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe05855a-7e8e-49cb-95f1-f57089c33150 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:21.473899 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-fe05855a-7e8e-49cb-95f1-f57089c33150 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:21.475173 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fe05855a-7e8e-49cb-95f1-f57089c33150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:21.483803 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 981/1962] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:49:21.488013 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15156d7-5681-41ca-b7fb-579ae5ea5260 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:21.498215 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15156d7-5681-41ca-b7fb-579ae5ea5260 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:21.503346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ccf85c-d0f7-4b12-81cd-bb55f0a4637e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:21.503961 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ccf85c-d0f7-4b12-81cd-bb55f0a4637e admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:21.504243 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ccf85c-d0f7-4b12-81cd-bb55f0a4637e admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:21.504517 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ccf85c-d0f7-4b12-81cd-bb55f0a4637e admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:21.505163 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ccf85c-d0f7-4b12-81cd-bb55f0a4637e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:49:21.505471 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ccf85c-d0f7-4b12-81cd-bb55f0a4637e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:21.507173 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ccf85c-d0f7-4b12-81cd-bb55f0a4637e admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:21.519112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 982/1963] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:49:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:49:21.523439 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe05855a-7e8e-49cb-95f1-f57089c33150 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:21.533286 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe05855a-7e8e-49cb-95f1-f57089c33150 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:21.537882 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e6170d-581f-4b63-9504-c16b6b76eb2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:21.538550 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e6170d-581f-4b63-9504-c16b6b76eb2f admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:21.538838 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e6170d-581f-4b63-9504-c16b6b76eb2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:21.539150 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e6170d-581f-4b63-9504-c16b6b76eb2f admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:21.539792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e6170d-581f-4b63-9504-c16b6b76eb2f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:21.542231 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e6170d-581f-4b63-9504-c16b6b76eb2f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:21.705318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 982/1964] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:49:21 2022] POST /identity/v3/projects => generated 373 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:21.707590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ccf85c-d0f7-4b12-81cd-bb55f0a4637e admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:21.719664 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ccf85c-d0f7-4b12-81cd-bb55f0a4637e admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:21.788274 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dcf365a-60ef-412b-ba5e-c98c6efe2ba7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:21.788806 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dcf365a-60ef-412b-ba5e-c98c6efe2ba7 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:21.789083 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dcf365a-60ef-412b-ba5e-c98c6efe2ba7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:21.789358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dcf365a-60ef-412b-ba5e-c98c6efe2ba7 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:21.789984 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dcf365a-60ef-412b-ba5e-c98c6efe2ba7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:21.791957 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dcf365a-60ef-412b-ba5e-c98c6efe2ba7 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:21.901918 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 983/1965] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:49:21 2022] POST /identity/v3/users => generated 287 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:21.906928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e6170d-581f-4b63-9504-c16b6b76eb2f admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:21.917038 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e6170d-581f-4b63-9504-c16b6b76eb2f admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:21.923117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a492d7-6841-484e-86c8-6bf67dd11160 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:21.923725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a492d7-6841-484e-86c8-6bf67dd11160 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:21.924008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a492d7-6841-484e-86c8-6bf67dd11160 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:21.924284 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a492d7-6841-484e-86c8-6bf67dd11160 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:21.925032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a492d7-6841-484e-86c8-6bf67dd11160 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:21.926548 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a492d7-6841-484e-86c8-6bf67dd11160 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:21.937739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 983/1966] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:21 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:49:21.943156 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dcf365a-60ef-412b-ba5e-c98c6efe2ba7 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:21.954625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dcf365a-60ef-412b-ba5e-c98c6efe2ba7 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:21.959674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d6c80c-75e6-44e0-902b-67f0497659a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:21.960276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d6c80c-75e6-44e0-902b-67f0497659a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:21.960555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d6c80c-75e6-44e0-902b-67f0497659a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:21.960826 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d6c80c-75e6-44e0-902b-67f0497659a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:21.961465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d6c80c-75e6-44e0-902b-67f0497659a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:21.963136 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d6c80c-75e6-44e0-902b-67f0497659a7 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:21.974287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 984/1967] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:21 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:21.978352 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a492d7-6841-484e-86c8-6bf67dd11160 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:21.988198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a492d7-6841-484e-86c8-6bf67dd11160 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:21.992816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332ed597-4ee2-4dd8-953b-da69d7818769 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:21.993463 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332ed597-4ee2-4dd8-953b-da69d7818769 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:21.993743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332ed597-4ee2-4dd8-953b-da69d7818769 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:21.994019 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332ed597-4ee2-4dd8-953b-da69d7818769 admin admin] PATH_INFO: `/v3/projects/37a1aef0396c463d8a10f0ee811e1f2a/users/ede7a69d69084b3583433547473532be/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:22.009884 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332ed597-4ee2-4dd8-953b-da69d7818769 admin admin] RBAC: Authorizing `identity:create_grant(project_id=37a1aef0396c463d8a10f0ee811e1f2a, user_id=ede7a69d69084b3583433547473532be, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:22.013894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332ed597-4ee2-4dd8-953b-da69d7818769 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:22.136451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 984/1968] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:21 2022] PUT /identity/v3/projects/37a1aef0396c463d8a10f0ee811e1f2a/users/ede7a69d69084b3583433547473532be/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:22.146724 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74430bd5-7aa1-4cc8-b29c-bc48a829ce74 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:22.147047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74430bd5-7aa1-4cc8-b29c-bc48a829ce74 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:22.147324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74430bd5-7aa1-4cc8-b29c-bc48a829ce74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:22.167393 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-74430bd5-7aa1-4cc8-b29c-bc48a829ce74 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:22.168678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-74430bd5-7aa1-4cc8-b29c-bc48a829ce74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:22.231528 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 985/1969] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:22.247672 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332ed597-4ee2-4dd8-953b-da69d7818769 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:22.250029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-332ed597-4ee2-4dd8-953b-da69d7818769 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:22.251213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:22.308056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332ed597-4ee2-4dd8-953b-da69d7818769 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:22.312293 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f467b217-f75c-4c36-9fd2-cbf36dfdac70 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:22.312906 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f467b217-f75c-4c36-9fd2-cbf36dfdac70 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:22.313190 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f467b217-f75c-4c36-9fd2-cbf36dfdac70 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:22.313464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f467b217-f75c-4c36-9fd2-cbf36dfdac70 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:22.322868 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f467b217-f75c-4c36-9fd2-cbf36dfdac70 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:22.325046 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f467b217-f75c-4c36-9fd2-cbf36dfdac70 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:22.335266 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 985/1970] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:22.594143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:24.645246 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:25.908137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:26.686248 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:27.682863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:28.029816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:29.136321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043381f-6a65-4034-b658-8d54075ad230 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:29.136851 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043381f-6a65-4034-b658-8d54075ad230 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:29.136851 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043381f-6a65-4034-b658-8d54075ad230 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:29.177190 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-5043381f-6a65-4034-b658-8d54075ad230 None None] MFA Rules not processed for user `ede7a69d69084b3583433547473532be`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:29.179420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-5043381f-6a65-4034-b658-8d54075ad230 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:29.193991 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:29.260108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 986/1971] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:29 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:29.264725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f467b217-f75c-4c36-9fd2-cbf36dfdac70 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:29.274409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f467b217-f75c-4c36-9fd2-cbf36dfdac70 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:29.279695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4496223-2e66-41eb-87f8-8fbb92da15ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:29.280296 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4496223-2e66-41eb-87f8-8fbb92da15ed admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:29.280583 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4496223-2e66-41eb-87f8-8fbb92da15ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:29.280859 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4496223-2e66-41eb-87f8-8fbb92da15ed admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:29.281494 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4496223-2e66-41eb-87f8-8fbb92da15ed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:29.283432 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4496223-2e66-41eb-87f8-8fbb92da15ed admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:29.372269 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 986/1972] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:49:29 2022] POST /identity/v3/projects => generated 371 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:29.376002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5043381f-6a65-4034-b658-8d54075ad230 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:29.386680 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5043381f-6a65-4034-b658-8d54075ad230 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:29.445211 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4af7a4-f741-4799-b897-e2c93f95bca5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:29.445977 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4af7a4-f741-4799-b897-e2c93f95bca5 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:29.446292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4af7a4-f741-4799-b897-e2c93f95bca5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:29.446570 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4af7a4-f741-4799-b897-e2c93f95bca5 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:29.447258 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4af7a4-f741-4799-b897-e2c93f95bca5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:29.449196 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4af7a4-f741-4799-b897-e2c93f95bca5 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:29.478840 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:29.559511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 987/1973] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:49:29 2022] POST /identity/v3/users => generated 292 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:29.563633 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4496223-2e66-41eb-87f8-8fbb92da15ed admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:29.573741 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4496223-2e66-41eb-87f8-8fbb92da15ed admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:29.579202 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe274e42-83db-41c5-aa54-6d3101720f56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:29.579816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe274e42-83db-41c5-aa54-6d3101720f56 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:29.580098 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe274e42-83db-41c5-aa54-6d3101720f56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:29.580374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe274e42-83db-41c5-aa54-6d3101720f56 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:29.581045 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe274e42-83db-41c5-aa54-6d3101720f56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:29.582668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe274e42-83db-41c5-aa54-6d3101720f56 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:29.593397 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 987/1974] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:29 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:29.597587 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4af7a4-f741-4799-b897-e2c93f95bca5 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:29.607626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4af7a4-f741-4799-b897-e2c93f95bca5 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:29.612214 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808ac4ef-eec2-4013-a545-687c31abf3c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:29.612863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808ac4ef-eec2-4013-a545-687c31abf3c0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:29.618420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808ac4ef-eec2-4013-a545-687c31abf3c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:29.618420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808ac4ef-eec2-4013-a545-687c31abf3c0 admin admin] PATH_INFO: `/v3/projects/80a5db390a074d4493fd3b17297ac181/users/8b849355226c45208f80839623d3db1e/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:29.627917 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808ac4ef-eec2-4013-a545-687c31abf3c0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=80a5db390a074d4493fd3b17297ac181, user_id=8b849355226c45208f80839623d3db1e, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:29.630546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808ac4ef-eec2-4013-a545-687c31abf3c0 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:29.708872 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 988/1975] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:29 2022] PUT /identity/v3/projects/80a5db390a074d4493fd3b17297ac181/users/8b849355226c45208f80839623d3db1e/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:29.713620 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe274e42-83db-41c5-aa54-6d3101720f56 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:29.724481 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe274e42-83db-41c5-aa54-6d3101720f56 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:29.782339 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5588764f-2eae-45f3-a02c-413d8d43be9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:29.782983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5588764f-2eae-45f3-a02c-413d8d43be9e admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:29.783264 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5588764f-2eae-45f3-a02c-413d8d43be9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:29.783542 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5588764f-2eae-45f3-a02c-413d8d43be9e admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:29.784179 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5588764f-2eae-45f3-a02c-413d8d43be9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:29.785793 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5588764f-2eae-45f3-a02c-413d8d43be9e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:29.797154 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 988/1976] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:29 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:29.801569 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808ac4ef-eec2-4013-a545-687c31abf3c0 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:29.812963 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808ac4ef-eec2-4013-a545-687c31abf3c0 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:29.817671 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738c5938-1ac9-4eb0-8ef8-97e42b547350 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:29.818329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738c5938-1ac9-4eb0-8ef8-97e42b547350 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:29.818612 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738c5938-1ac9-4eb0-8ef8-97e42b547350 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:29.818888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738c5938-1ac9-4eb0-8ef8-97e42b547350 admin admin] PATH_INFO: `/v3/projects/80a5db390a074d4493fd3b17297ac181/users/8b849355226c45208f80839623d3db1e/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:29.820932 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738c5938-1ac9-4eb0-8ef8-97e42b547350 admin admin] RBAC: Authorizing `identity:create_grant(project_id=80a5db390a074d4493fd3b17297ac181, user_id=8b849355226c45208f80839623d3db1e, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:29.823583 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738c5938-1ac9-4eb0-8ef8-97e42b547350 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:29.836335 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 989/1977] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:29 2022] PUT /identity/v3/projects/80a5db390a074d4493fd3b17297ac181/users/8b849355226c45208f80839623d3db1e/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:49:30.559336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:31.846527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:32.159479 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:35.348326 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=6,keystone:INSERT=5 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:35.764780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:36.408950 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2335bebb-5b70-4d44-a75f-b72753da0ed2 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:36.409241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2335bebb-5b70-4d44-a75f-b72753da0ed2 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:36.410386 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2335bebb-5b70-4d44-a75f-b72753da0ed2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:36.455732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2335bebb-5b70-4d44-a75f-b72753da0ed2 None None] MFA Rules not processed for user `8b849355226c45208f80839623d3db1e`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:36.460545 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2335bebb-5b70-4d44-a75f-b72753da0ed2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:36.581326 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 989/1978] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:36 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:36.595940 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738c5938-1ac9-4eb0-8ef8-97e42b547350 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:36.598534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-738c5938-1ac9-4eb0-8ef8-97e42b547350 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:36.696464 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738c5938-1ac9-4eb0-8ef8-97e42b547350 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:36.702508 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd72db9-87e5-4e03-8ecf-8414aee47fbb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:36.703158 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd72db9-87e5-4e03-8ecf-8414aee47fbb service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:36.703443 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd72db9-87e5-4e03-8ecf-8414aee47fbb service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:36.703747 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd72db9-87e5-4e03-8ecf-8414aee47fbb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:36.761894 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd72db9-87e5-4e03-8ecf-8414aee47fbb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:36.763584 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd72db9-87e5-4e03-8ecf-8414aee47fbb service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:36.772908 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 990/1979] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:36 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:38.433544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2335bebb-5b70-4d44-a75f-b72753da0ed2 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:38.436431 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2335bebb-5b70-4d44-a75f-b72753da0ed2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:38.514566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:38.517081 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2335bebb-5b70-4d44-a75f-b72753da0ed2 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:38.522800 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc58d41e-1a50-42ed-8dbe-5ae1faf5e817 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:38.523535 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc58d41e-1a50-42ed-8dbe-5ae1faf5e817 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:38.523817 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc58d41e-1a50-42ed-8dbe-5ae1faf5e817 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:38.524092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc58d41e-1a50-42ed-8dbe-5ae1faf5e817 admin admin] PATH_INFO: `/v3/users/7547f6b8678c4f308608f6146ffb5151` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:38.525475 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc58d41e-1a50-42ed-8dbe-5ae1faf5e817 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7547f6b8678c4f308608f6146ffb5151)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:38.527618 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc58d41e-1a50-42ed-8dbe-5ae1faf5e817 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:38.784439 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:38.802899 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fc58d41e-1a50-42ed-8dbe-5ae1faf5e817 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7547f6b8678c4f308608f6146ffb5151'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:38.828346 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fc58d41e-1a50-42ed-8dbe-5ae1faf5e817 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7547f6b8678c4f308608f6146ffb5151'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:38.937076 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fc58d41e-1a50-42ed-8dbe-5ae1faf5e817 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7547f6b8678c4f308608f6146ffb5151'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:38.938135 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fc58d41e-1a50-42ed-8dbe-5ae1faf5e817 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7547f6b8678c4f308608f6146ffb5151'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:38.949893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fc58d41e-1a50-42ed-8dbe-5ae1faf5e817 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7547f6b8678c4f308608f6146ffb5151'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:38.959057 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 990/1980] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:49:38 2022] DELETE /identity/v3/users/7547f6b8678c4f308608f6146ffb5151 => generated 0 bytes in 526 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:38.963837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243dc8ec-1b5d-497c-b93e-ba4cd67878e5 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:38.964121 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243dc8ec-1b5d-497c-b93e-ba4cd67878e5 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:38.964393 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243dc8ec-1b5d-497c-b93e-ba4cd67878e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:38.991384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-243dc8ec-1b5d-497c-b93e-ba4cd67878e5 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:38.992679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-243dc8ec-1b5d-497c-b93e-ba4cd67878e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:39.113794 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 991/1981] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:39.126343 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc58d41e-1a50-42ed-8dbe-5ae1faf5e817 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:39.127925 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fc58d41e-1a50-42ed-8dbe-5ae1faf5e817 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:39.205346 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc58d41e-1a50-42ed-8dbe-5ae1faf5e817 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:39.211825 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6046058-d5b4-40c3-8688-06887fedf194 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:39.212468 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6046058-d5b4-40c3-8688-06887fedf194 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:39.212741 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6046058-d5b4-40c3-8688-06887fedf194 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:39.213341 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6046058-d5b4-40c3-8688-06887fedf194 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:39.230375 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6046058-d5b4-40c3-8688-06887fedf194 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:39.232783 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6046058-d5b4-40c3-8688-06887fedf194 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:39.259967 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 991/1982] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:39.821937 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:39.869748 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243dc8ec-1b5d-497c-b93e-ba4cd67878e5 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:39.872926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-243dc8ec-1b5d-497c-b93e-ba4cd67878e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:39.900221 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243dc8ec-1b5d-497c-b93e-ba4cd67878e5 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:39.928532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcadc081-41f1-4899-a36d-39d9348f0809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:39.928532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcadc081-41f1-4899-a36d-39d9348f0809 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:39.928532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcadc081-41f1-4899-a36d-39d9348f0809 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:39.928947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcadc081-41f1-4899-a36d-39d9348f0809 admin admin] PATH_INFO: `/v3/projects/10d20578af134f4fa99d5b3479ef1920` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:39.930346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcadc081-41f1-4899-a36d-39d9348f0809 admin admin] RBAC: Authorizing `identity:delete_project(project_id=10d20578af134f4fa99d5b3479ef1920)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:39.933548 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcadc081-41f1-4899-a36d-39d9348f0809 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:40.081444 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-fcadc081-41f1-4899-a36d-39d9348f0809 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '10d20578af134f4fa99d5b3479ef1920', 'request_id': 'req-fcadc081-41f1-4899-a36d-39d9348f0809'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:40.193470 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-fcadc081-41f1-4899-a36d-39d9348f0809 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '10d20578af134f4fa99d5b3479ef1920', 'request_id': 'req-fcadc081-41f1-4899-a36d-39d9348f0809'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:40.202846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-fcadc081-41f1-4899-a36d-39d9348f0809 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '10d20578af134f4fa99d5b3479ef1920', 'request_id': 'req-fcadc081-41f1-4899-a36d-39d9348f0809'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:40.235129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-fcadc081-41f1-4899-a36d-39d9348f0809 admin admin] The token cache is being invalidate because project 10d20578af134f4fa99d5b3479ef1920 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:49:40.235430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-fcadc081-41f1-4899-a36d-39d9348f0809 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:40.236401 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-fcadc081-41f1-4899-a36d-39d9348f0809 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:40.241326 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 992/1983] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:49:39 2022] DELETE /identity/v3/projects/10d20578af134f4fa99d5b3479ef1920 => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:40.246409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b038d9ab-b70b-44b8-9894-61ca9b4ceaa8 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:40.249862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b038d9ab-b70b-44b8-9894-61ca9b4ceaa8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:40.250155 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b038d9ab-b70b-44b8-9894-61ca9b4ceaa8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:40.291717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-b038d9ab-b70b-44b8-9894-61ca9b4ceaa8 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:40.294790 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b038d9ab-b70b-44b8-9894-61ca9b4ceaa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:40.408295 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 992/1984] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:40.410096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cb91ab-d923-429c-bb4b-89f036d0c762 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:40.410383 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cb91ab-d923-429c-bb4b-89f036d0c762 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:40.410660 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cb91ab-d923-429c-bb4b-89f036d0c762 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:40.436420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-47cb91ab-d923-429c-bb4b-89f036d0c762 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:40.437790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-47cb91ab-d923-429c-bb4b-89f036d0c762 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:40.448939 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 993/1985] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:40.452564 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b038d9ab-b70b-44b8-9894-61ca9b4ceaa8 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:40.462814 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b038d9ab-b70b-44b8-9894-61ca9b4ceaa8 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:40.471052 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb30c7c-8bfa-4b1f-8b8c-f84f73403a5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:40.472233 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb30c7c-8bfa-4b1f-8b8c-f84f73403a5c admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:40.472673 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb30c7c-8bfa-4b1f-8b8c-f84f73403a5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:40.476366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb30c7c-8bfa-4b1f-8b8c-f84f73403a5c admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:40.476366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb30c7c-8bfa-4b1f-8b8c-f84f73403a5c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:49:40.476366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb30c7c-8bfa-4b1f-8b8c-f84f73403a5c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:40.476890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb30c7c-8bfa-4b1f-8b8c-f84f73403a5c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:40.493434 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4055fc-a97f-45ac-ade5-76427bdf638c None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:40.494144 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4055fc-a97f-45ac-ade5-76427bdf638c None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:40.494835 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4055fc-a97f-45ac-ade5-76427bdf638c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:40.501284 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 993/1986] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:49:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:49:40.502268 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb30c7c-8bfa-4b1f-8b8c-f84f73403a5c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:40.521420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb30c7c-8bfa-4b1f-8b8c-f84f73403a5c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:40.528158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0fe231-2079-4337-aaf7-b48c827dc570 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:40.529770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0fe231-2079-4337-aaf7-b48c827dc570 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:40.530861 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0fe231-2079-4337-aaf7-b48c827dc570 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:40.531489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-2b4055fc-a97f-45ac-ade5-76427bdf638c None None] MFA Rules not processed for user `ede7a69d69084b3583433547473532be`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:40.532605 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0fe231-2079-4337-aaf7-b48c827dc570 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:40.533807 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0fe231-2079-4337-aaf7-b48c827dc570 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:40.536048 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-2b4055fc-a97f-45ac-ade5-76427bdf638c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:40.539481 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0fe231-2079-4337-aaf7-b48c827dc570 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:40.608464 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 994/1987] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:40 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:49:40.706360 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 994/1988] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:49:40 2022] POST /identity/v3/projects => generated 369 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:40.707803 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4055fc-a97f-45ac-ade5-76427bdf638c None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:40.725173 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4055fc-a97f-45ac-ade5-76427bdf638c None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:40.812683 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451553b1-6400-4ff5-8c7d-95850d997052 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:40.813333 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451553b1-6400-4ff5-8c7d-95850d997052 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:40.813610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451553b1-6400-4ff5-8c7d-95850d997052 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:40.813882 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451553b1-6400-4ff5-8c7d-95850d997052 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:40.817144 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451553b1-6400-4ff5-8c7d-95850d997052 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:40.817144 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451553b1-6400-4ff5-8c7d-95850d997052 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:40.833974 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f951a29c-6501-4020-9194-9ce6405b66e8 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:40.834511 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f951a29c-6501-4020-9194-9ce6405b66e8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:40.834841 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f951a29c-6501-4020-9194-9ce6405b66e8 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:40.836825 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 995/1989] 10.222.0.49 () {54 vars in 921 bytes} [Fri Sep 16 03:49:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 03:49:40.845712 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7c752e-dda5-4c1b-a186-e2d54e7bf75a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:40.846090 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7c752e-dda5-4c1b-a186-e2d54e7bf75a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:40.846419 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7c752e-dda5-4c1b-a186-e2d54e7bf75a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:40.882444 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7b7c752e-dda5-4c1b-a186-e2d54e7bf75a None None] MFA Rules not processed for user `176da4679fcb46148da9289fccdad5c2`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:40.883828 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7b7c752e-dda5-4c1b-a186-e2d54e7bf75a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:40.970543 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 996/1990] 10.222.0.49 () {58 vars in 1018 bytes} [Fri Sep 16 03:49:40 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:49:40.985762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7c752e-dda5-4c1b-a186-e2d54e7bf75a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:40.986286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 995/1991] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:49:40 2022] POST /identity/v3/users => generated 285 bytes in 279 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:40.987842 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7b7c752e-dda5-4c1b-a186-e2d54e7bf75a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:40.990770 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451553b1-6400-4ff5-8c7d-95850d997052 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:41.015174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451553b1-6400-4ff5-8c7d-95850d997052 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:41.018867 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7c752e-dda5-4c1b-a186-e2d54e7bf75a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:41.023866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e649561a-4020-45ee-822a-e7bc1dad6350 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:41.025208 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e649561a-4020-45ee-822a-e7bc1dad6350 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:41.025875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e649561a-4020-45ee-822a-e7bc1dad6350 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:41.026537 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e649561a-4020-45ee-822a-e7bc1dad6350 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:41.027916 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e649561a-4020-45ee-822a-e7bc1dad6350 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:41.035641 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7143e7ae-a3b7-4516-89e9-f06ade055200 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '176da4679fcb46148da9289fccdad5c2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:41.036533 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7143e7ae-a3b7-4516-89e9-f06ade055200 service placement] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:41.036888 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7143e7ae-a3b7-4516-89e9-f06ade055200 service placement] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:41.037296 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7143e7ae-a3b7-4516-89e9-f06ade055200 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:41.043034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e649561a-4020-45ee-822a-e7bc1dad6350 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:41.057938 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7143e7ae-a3b7-4516-89e9-f06ade055200 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:41.060617 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7143e7ae-a3b7-4516-89e9-f06ade055200 service placement] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:41.062320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 996/1992] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:40 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:49:41.066474 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e649561a-4020-45ee-822a-e7bc1dad6350 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:41.079995 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 997/1993] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:40 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:49:41.081698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e649561a-4020-45ee-822a-e7bc1dad6350 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:41.087342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7979796-9744-44e8-8ebb-0c8ed08cccd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:41.088073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7979796-9744-44e8-8ebb-0c8ed08cccd6 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:41.088520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7979796-9744-44e8-8ebb-0c8ed08cccd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:41.088984 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7979796-9744-44e8-8ebb-0c8ed08cccd6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:41.089867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7979796-9744-44e8-8ebb-0c8ed08cccd6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:41.092900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7979796-9744-44e8-8ebb-0c8ed08cccd6 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:41.108410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 997/1994] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:41 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:41.112295 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7143e7ae-a3b7-4516-89e9-f06ade055200 service placement] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:41.126538 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7143e7ae-a3b7-4516-89e9-f06ade055200 service placement] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:41.132796 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50363b6d-484f-46d4-b4db-33ba9214d474 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:41.133463 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50363b6d-484f-46d4-b4db-33ba9214d474 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:41.133869 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50363b6d-484f-46d4-b4db-33ba9214d474 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:41.134406 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50363b6d-484f-46d4-b4db-33ba9214d474 admin admin] PATH_INFO: `/v3/projects/f55241ec40414308b1b8fc02618a8188/users/93af2ba9dd234e6eac7cf25f0be68f9b/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:41.148668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:41.153317 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50363b6d-484f-46d4-b4db-33ba9214d474 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f55241ec40414308b1b8fc02618a8188, user_id=93af2ba9dd234e6eac7cf25f0be68f9b, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:41.156176 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50363b6d-484f-46d4-b4db-33ba9214d474 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:41.346934 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 998/1995] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:41 2022] PUT /identity/v3/projects/f55241ec40414308b1b8fc02618a8188/users/93af2ba9dd234e6eac7cf25f0be68f9b/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:41.359065 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738ddf16-7988-4ffc-828e-effbeaada478 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:41.359357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738ddf16-7988-4ffc-828e-effbeaada478 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:41.359637 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738ddf16-7988-4ffc-828e-effbeaada478 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:41.388923 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-738ddf16-7988-4ffc-828e-effbeaada478 None None] MFA Rules not processed for user `93af2ba9dd234e6eac7cf25f0be68f9b`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:41.390148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-738ddf16-7988-4ffc-828e-effbeaada478 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:41.458784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 998/1996] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:41 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:49:41.470762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50363b6d-484f-46d4-b4db-33ba9214d474 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:41.472705 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-50363b6d-484f-46d4-b4db-33ba9214d474 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:41.534121 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50363b6d-484f-46d4-b4db-33ba9214d474 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:41.537876 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8770281-2104-404c-8e61-5b54b2470035 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ceb9fc31c345ccabaee8f30a3982a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:41.538492 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8770281-2104-404c-8e61-5b54b2470035 service glance] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:41.538795 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8770281-2104-404c-8e61-5b54b2470035 service glance] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:41.539111 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8770281-2104-404c-8e61-5b54b2470035 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:41.549516 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8770281-2104-404c-8e61-5b54b2470035 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:41.552442 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8770281-2104-404c-8e61-5b54b2470035 service glance] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:41.564245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 999/1997] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:41 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:41.899126 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738ddf16-7988-4ffc-828e-effbeaada478 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:41.900669 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-738ddf16-7988-4ffc-828e-effbeaada478 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:41.967082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738ddf16-7988-4ffc-828e-effbeaada478 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:41.971704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d7eadc-a224-475a-a884-3720ad547007 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:41.972422 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d7eadc-a224-475a-a884-3720ad547007 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:41.972807 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d7eadc-a224-475a-a884-3720ad547007 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:41.973167 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d7eadc-a224-475a-a884-3720ad547007 admin admin] PATH_INFO: `/v3/users/bc5eb4c34a614860921f281fc7d4413c` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:41.974370 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d7eadc-a224-475a-a884-3720ad547007 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bc5eb4c34a614860921f281fc7d4413c)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:41.976422 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d7eadc-a224-475a-a884-3720ad547007 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:42.256295 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a1d7eadc-a224-475a-a884-3720ad547007 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bc5eb4c34a614860921f281fc7d4413c'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:42.376740 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:42.378872 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a1d7eadc-a224-475a-a884-3720ad547007 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bc5eb4c34a614860921f281fc7d4413c'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:42.399745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a1d7eadc-a224-475a-a884-3720ad547007 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bc5eb4c34a614860921f281fc7d4413c'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:42.400477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a1d7eadc-a224-475a-a884-3720ad547007 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bc5eb4c34a614860921f281fc7d4413c'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:42.409951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a1d7eadc-a224-475a-a884-3720ad547007 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bc5eb4c34a614860921f281fc7d4413c'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:42.427754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 999/1998] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:49:41 2022] DELETE /identity/v3/users/bc5eb4c34a614860921f281fc7d4413c => generated 0 bytes in 529 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:42.753658 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:42.756476 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8770281-2104-404c-8e61-5b54b2470035 service glance] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:42.776086 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8770281-2104-404c-8e61-5b54b2470035 service glance] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:42.835255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d290aa-ae81-4053-a8dd-4ff46db35f3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:42.836019 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d290aa-ae81-4053-a8dd-4ff46db35f3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:42.836586 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d290aa-ae81-4053-a8dd-4ff46db35f3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:42.836867 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d290aa-ae81-4053-a8dd-4ff46db35f3a admin admin] PATH_INFO: `/v3/projects/bed14ff70a404f71a3e7eb574ea3a8cb` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:42.838329 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d290aa-ae81-4053-a8dd-4ff46db35f3a admin admin] RBAC: Authorizing `identity:delete_project(project_id=bed14ff70a404f71a3e7eb574ea3a8cb)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:42.840452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d290aa-ae81-4053-a8dd-4ff46db35f3a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:43.007356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-43d290aa-ae81-4053-a8dd-4ff46db35f3a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bed14ff70a404f71a3e7eb574ea3a8cb', 'request_id': 'req-43d290aa-ae81-4053-a8dd-4ff46db35f3a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:43.013889 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-43d290aa-ae81-4053-a8dd-4ff46db35f3a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bed14ff70a404f71a3e7eb574ea3a8cb', 'request_id': 'req-43d290aa-ae81-4053-a8dd-4ff46db35f3a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:43.096340 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-43d290aa-ae81-4053-a8dd-4ff46db35f3a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bed14ff70a404f71a3e7eb574ea3a8cb', 'request_id': 'req-43d290aa-ae81-4053-a8dd-4ff46db35f3a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:43.108822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-43d290aa-ae81-4053-a8dd-4ff46db35f3a admin admin] The token cache is being invalidate because project bed14ff70a404f71a3e7eb574ea3a8cb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:49:43.109152 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-43d290aa-ae81-4053-a8dd-4ff46db35f3a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:43.110566 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-43d290aa-ae81-4053-a8dd-4ff46db35f3a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:43.112090 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1000/1999] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:49:42 2022] DELETE /identity/v3/projects/bed14ff70a404f71a3e7eb574ea3a8cb => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:43.116108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d7eadc-a224-475a-a884-3720ad547007 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:43.117722 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a1d7eadc-a224-475a-a884-3720ad547007 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:43.132737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d7eadc-a224-475a-a884-3720ad547007 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:43.197289 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee3e4f8-e23d-4f7c-9257-fd683382e21a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:43.197925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee3e4f8-e23d-4f7c-9257-fd683382e21a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:43.198215 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee3e4f8-e23d-4f7c-9257-fd683382e21a admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:43.198557 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee3e4f8-e23d-4f7c-9257-fd683382e21a admin admin] PATH_INFO: `/v3/users/342b2b7e7d0e4262b344aa00c57c11cc` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:43.199733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee3e4f8-e23d-4f7c-9257-fd683382e21a admin admin] RBAC: Authorizing `identity:delete_user(user_id=342b2b7e7d0e4262b344aa00c57c11cc)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:43.201868 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee3e4f8-e23d-4f7c-9257-fd683382e21a admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:43.546435 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-cee3e4f8-e23d-4f7c-9257-fd683382e21a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '342b2b7e7d0e4262b344aa00c57c11cc'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:43.677603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-cee3e4f8-e23d-4f7c-9257-fd683382e21a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '342b2b7e7d0e4262b344aa00c57c11cc'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:43.690004 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-cee3e4f8-e23d-4f7c-9257-fd683382e21a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '342b2b7e7d0e4262b344aa00c57c11cc'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:43.690634 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-cee3e4f8-e23d-4f7c-9257-fd683382e21a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '342b2b7e7d0e4262b344aa00c57c11cc'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:43.697541 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-cee3e4f8-e23d-4f7c-9257-fd683382e21a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '342b2b7e7d0e4262b344aa00c57c11cc'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:43.705281 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1000/2000] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:49:43 2022] DELETE /identity/v3/users/342b2b7e7d0e4262b344aa00c57c11cc => generated 0 bytes in 589 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:44.013673 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d290aa-ae81-4053-a8dd-4ff46db35f3a admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:44.015275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-43d290aa-ae81-4053-a8dd-4ff46db35f3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:44.030281 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d290aa-ae81-4053-a8dd-4ff46db35f3a admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:44.080422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:44.096073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f317361d-b590-4e28-9758-a2034936f941 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:44.096711 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f317361d-b590-4e28-9758-a2034936f941 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:44.097015 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f317361d-b590-4e28-9758-a2034936f941 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:44.097271 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f317361d-b590-4e28-9758-a2034936f941 admin admin] PATH_INFO: `/v3/projects/f3ceb13de3b94aec80ec802c6a5061f1` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:44.098415 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f317361d-b590-4e28-9758-a2034936f941 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f3ceb13de3b94aec80ec802c6a5061f1)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:44.100555 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f317361d-b590-4e28-9758-a2034936f941 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:44.238804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f317361d-b590-4e28-9758-a2034936f941 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f3ceb13de3b94aec80ec802c6a5061f1', 'request_id': 'req-f317361d-b590-4e28-9758-a2034936f941'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:44.246025 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f317361d-b590-4e28-9758-a2034936f941 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f3ceb13de3b94aec80ec802c6a5061f1', 'request_id': 'req-f317361d-b590-4e28-9758-a2034936f941'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:44.370521 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f317361d-b590-4e28-9758-a2034936f941 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f3ceb13de3b94aec80ec802c6a5061f1', 'request_id': 'req-f317361d-b590-4e28-9758-a2034936f941'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:44.381884 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f317361d-b590-4e28-9758-a2034936f941 admin admin] The token cache is being invalidate because project f3ceb13de3b94aec80ec802c6a5061f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:49:44.382364 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f317361d-b590-4e28-9758-a2034936f941 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:44.383033 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f317361d-b590-4e28-9758-a2034936f941 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:44.384828 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1001/2001] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:49:44 2022] DELETE /identity/v3/projects/f3ceb13de3b94aec80ec802c6a5061f1 => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:44.390238 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475959bf-fe7e-4c7b-8591-6f0e796d08cd None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:44.390600 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475959bf-fe7e-4c7b-8591-6f0e796d08cd None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:44.390977 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475959bf-fe7e-4c7b-8591-6f0e796d08cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:44.412945 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-475959bf-fe7e-4c7b-8591-6f0e796d08cd None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:44.414413 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-475959bf-fe7e-4c7b-8591-6f0e796d08cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:44.481425 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1001/2002] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:49:44.490147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a29dd3-d8ee-4288-9267-dfdba0a61900 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:44.490549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a29dd3-d8ee-4288-9267-dfdba0a61900 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:44.491065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a29dd3-d8ee-4288-9267-dfdba0a61900 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:44.513514 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-78a29dd3-d8ee-4288-9267-dfdba0a61900 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:44.515160 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-78a29dd3-d8ee-4288-9267-dfdba0a61900 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:44.524225 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1002/2003] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:44.530128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475959bf-fe7e-4c7b-8591-6f0e796d08cd None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:44.540607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475959bf-fe7e-4c7b-8591-6f0e796d08cd None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:44.545408 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1e91a4-9537-4d6d-b400-ae4d1605d17c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:44.546038 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1e91a4-9537-4d6d-b400-ae4d1605d17c admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:44.546319 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1e91a4-9537-4d6d-b400-ae4d1605d17c admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:44.546592 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1e91a4-9537-4d6d-b400-ae4d1605d17c admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:44.549831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1e91a4-9537-4d6d-b400-ae4d1605d17c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:49:44.549831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1e91a4-9537-4d6d-b400-ae4d1605d17c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:44.549831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1e91a4-9537-4d6d-b400-ae4d1605d17c admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:44.562353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1002/2004] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:49:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:49:44.566970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a29dd3-d8ee-4288-9267-dfdba0a61900 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:44.574963 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:44.578424 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a29dd3-d8ee-4288-9267-dfdba0a61900 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:44.583517 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3783aa-8f7d-48dc-8372-c9e79dbb33eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:44.584057 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3783aa-8f7d-48dc-8372-c9e79dbb33eb admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:44.584389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3783aa-8f7d-48dc-8372-c9e79dbb33eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:44.584810 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3783aa-8f7d-48dc-8372-c9e79dbb33eb admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:44.585756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3783aa-8f7d-48dc-8372-c9e79dbb33eb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:44.587694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3783aa-8f7d-48dc-8372-c9e79dbb33eb admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:44.704638 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1003/2005] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:49:44 2022] POST /identity/v3/projects => generated 375 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:44.709927 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1e91a4-9537-4d6d-b400-ae4d1605d17c admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:44.726790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1e91a4-9537-4d6d-b400-ae4d1605d17c admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:44.833254 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10c64e0-3e35-42b0-b825-ba4f388bee88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:44.836571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10c64e0-3e35-42b0-b825-ba4f388bee88 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:44.836571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10c64e0-3e35-42b0-b825-ba4f388bee88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:44.836571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10c64e0-3e35-42b0-b825-ba4f388bee88 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:44.836571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10c64e0-3e35-42b0-b825-ba4f388bee88 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:44.837092 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10c64e0-3e35-42b0-b825-ba4f388bee88 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:44.958782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1003/2006] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:49:44 2022] POST /identity/v3/users => generated 288 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:44.962963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3783aa-8f7d-48dc-8372-c9e79dbb33eb admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:44.973388 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3783aa-8f7d-48dc-8372-c9e79dbb33eb admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:44.978157 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4959a6bb-cc4a-4970-a241-c8603534c0ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:44.978840 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4959a6bb-cc4a-4970-a241-c8603534c0ca admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:44.979217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4959a6bb-cc4a-4970-a241-c8603534c0ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:44.979581 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4959a6bb-cc4a-4970-a241-c8603534c0ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:44.980374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4959a6bb-cc4a-4970-a241-c8603534c0ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:44.982079 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4959a6bb-cc4a-4970-a241-c8603534c0ca admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:44.993751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1004/2007] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:49:45.000065 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10c64e0-3e35-42b0-b825-ba4f388bee88 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:45.010036 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10c64e0-3e35-42b0-b825-ba4f388bee88 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:45.016856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0877cda3-bd65-4cf0-a029-fe87b398a542 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:45.017700 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0877cda3-bd65-4cf0-a029-fe87b398a542 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:45.018141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0877cda3-bd65-4cf0-a029-fe87b398a542 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:45.018484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0877cda3-bd65-4cf0-a029-fe87b398a542 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:45.019251 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0877cda3-bd65-4cf0-a029-fe87b398a542 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:45.021612 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0877cda3-bd65-4cf0-a029-fe87b398a542 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:45.037067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1004/2008] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:44 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:45.041345 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4959a6bb-cc4a-4970-a241-c8603534c0ca admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:45.053256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4959a6bb-cc4a-4970-a241-c8603534c0ca admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:45.060131 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae94974-1be8-47d9-ab84-1349757e5127 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:45.061092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae94974-1be8-47d9-ab84-1349757e5127 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:45.061092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae94974-1be8-47d9-ab84-1349757e5127 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:45.061388 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae94974-1be8-47d9-ab84-1349757e5127 admin admin] PATH_INFO: `/v3/projects/f71250c81e7c47d7ae28db633a23d6de/users/3deb8742f42c4569925fcf4a9eb50405/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:45.079525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae94974-1be8-47d9-ab84-1349757e5127 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f71250c81e7c47d7ae28db633a23d6de, user_id=3deb8742f42c4569925fcf4a9eb50405, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:45.083352 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae94974-1be8-47d9-ab84-1349757e5127 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:45.107976 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:45.216704 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1005/2009] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:45 2022] PUT /identity/v3/projects/f71250c81e7c47d7ae28db633a23d6de/users/3deb8742f42c4569925fcf4a9eb50405/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:49:45.228129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89a1839-76a8-4338-ac20-68d961dc83b8 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:45.228566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89a1839-76a8-4338-ac20-68d961dc83b8 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:45.228911 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89a1839-76a8-4338-ac20-68d961dc83b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:45.267191 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-b89a1839-76a8-4338-ac20-68d961dc83b8 None None] MFA Rules not processed for user `3deb8742f42c4569925fcf4a9eb50405`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:45.268516 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b89a1839-76a8-4338-ac20-68d961dc83b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:45.340450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1005/2010] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:45 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:45.351712 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae94974-1be8-47d9-ab84-1349757e5127 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:45.353517 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8ae94974-1be8-47d9-ab84-1349757e5127 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:45.415744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae94974-1be8-47d9-ab84-1349757e5127 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:45.422434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa0b30b-fbb9-4f7e-ae4a-c29f3a17d395 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:45.423362 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa0b30b-fbb9-4f7e-ae4a-c29f3a17d395 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:45.423671 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa0b30b-fbb9-4f7e-ae4a-c29f3a17d395 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:45.423827 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa0b30b-fbb9-4f7e-ae4a-c29f3a17d395 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:45.434488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa0b30b-fbb9-4f7e-ae4a-c29f3a17d395 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:45.437558 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa0b30b-fbb9-4f7e-ae4a-c29f3a17d395 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:45.447410 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1006/2011] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:45 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:45.525287 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:46.707666 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:46.913890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:48.177397 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:48.264494 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9345b7a2-6e85-422c-865a-6d820e776089 None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:48.264892 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9345b7a2-6e85-422c-865a-6d820e776089 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:48.265415 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9345b7a2-6e85-422c-865a-6d820e776089 None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:48.267094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1006/2012] 10.222.0.49 () {54 vars in 921 bytes} [Fri Sep 16 03:49:48 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:49:48.323288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9899e62a-8c55-45e7-9dd8-e1f22ab398c9 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:48.324179 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9899e62a-8c55-45e7-9dd8-e1f22ab398c9 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:48.324601 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9899e62a-8c55-45e7-9dd8-e1f22ab398c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:48.353002 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9899e62a-8c55-45e7-9dd8-e1f22ab398c9 None None] MFA Rules not processed for user `176da4679fcb46148da9289fccdad5c2`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:48.354206 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9899e62a-8c55-45e7-9dd8-e1f22ab398c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:48.367341 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:48.436372 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1007/2013] 10.222.0.49 () {58 vars in 1018 bytes} [Fri Sep 16 03:49:48 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:48.460357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9345b7a2-6e85-422c-865a-6d820e776089 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:48.462157 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9899e62a-8c55-45e7-9dd8-e1f22ab398c9 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:48.462643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-9345b7a2-6e85-422c-865a-6d820e776089 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:48.464082 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9899e62a-8c55-45e7-9dd8-e1f22ab398c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:48.491031 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9345b7a2-6e85-422c-865a-6d820e776089 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:48.506679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578453f6-54c5-4554-96de-2847ba9a1c85 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '176da4679fcb46148da9289fccdad5c2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:48.506679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578453f6-54c5-4554-96de-2847ba9a1c85 service placement] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:48.506679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578453f6-54c5-4554-96de-2847ba9a1c85 service placement] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:48.506679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578453f6-54c5-4554-96de-2847ba9a1c85 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:48.511528 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578453f6-54c5-4554-96de-2847ba9a1c85 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:48.523128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578453f6-54c5-4554-96de-2847ba9a1c85 service placement] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:48.523435 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1007/2014] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:48 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:48.545824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9899e62a-8c55-45e7-9dd8-e1f22ab398c9 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:48.550995 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb4cda3-deb9-4b72-8b0f-ed0f2b9de913 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:48.551716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb4cda3-deb9-4b72-8b0f-ed0f2b9de913 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:48.551993 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb4cda3-deb9-4b72-8b0f-ed0f2b9de913 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:48.552559 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb4cda3-deb9-4b72-8b0f-ed0f2b9de913 admin admin] PATH_INFO: `/v3/users/c1e4846769d44451b239dee5c892b5e8` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:48.553723 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb4cda3-deb9-4b72-8b0f-ed0f2b9de913 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c1e4846769d44451b239dee5c892b5e8)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:48.555664 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb4cda3-deb9-4b72-8b0f-ed0f2b9de913 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:48.804505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-aeb4cda3-deb9-4b72-8b0f-ed0f2b9de913 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c1e4846769d44451b239dee5c892b5e8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:48.816352 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-aeb4cda3-deb9-4b72-8b0f-ed0f2b9de913 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c1e4846769d44451b239dee5c892b5e8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:48.922465 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-aeb4cda3-deb9-4b72-8b0f-ed0f2b9de913 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c1e4846769d44451b239dee5c892b5e8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:48.923138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-aeb4cda3-deb9-4b72-8b0f-ed0f2b9de913 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c1e4846769d44451b239dee5c892b5e8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:48.933874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-aeb4cda3-deb9-4b72-8b0f-ed0f2b9de913 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c1e4846769d44451b239dee5c892b5e8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:48.942582 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1008/2015] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:49:48 2022] DELETE /identity/v3/users/c1e4846769d44451b239dee5c892b5e8 => generated 0 bytes in 481 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:49.240218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578453f6-54c5-4554-96de-2847ba9a1c85 service placement] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:49.257500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578453f6-54c5-4554-96de-2847ba9a1c85 service placement] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:49.317336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0353f43-439c-4cc4-9070-51cd19ca3ff4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:49.317899 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0353f43-439c-4cc4-9070-51cd19ca3ff4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:49.318271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0353f43-439c-4cc4-9070-51cd19ca3ff4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:49.318552 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0353f43-439c-4cc4-9070-51cd19ca3ff4 admin admin] PATH_INFO: `/v3/projects/d39c13d4012549c2890bf1d904d1c041` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:49.319713 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0353f43-439c-4cc4-9070-51cd19ca3ff4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d39c13d4012549c2890bf1d904d1c041)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:49.323477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0353f43-439c-4cc4-9070-51cd19ca3ff4 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:49.440272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d0353f43-439c-4cc4-9070-51cd19ca3ff4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd39c13d4012549c2890bf1d904d1c041', 'request_id': 'req-d0353f43-439c-4cc4-9070-51cd19ca3ff4'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:49.547256 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d0353f43-439c-4cc4-9070-51cd19ca3ff4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd39c13d4012549c2890bf1d904d1c041', 'request_id': 'req-d0353f43-439c-4cc4-9070-51cd19ca3ff4'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:49.555555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d0353f43-439c-4cc4-9070-51cd19ca3ff4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd39c13d4012549c2890bf1d904d1c041', 'request_id': 'req-d0353f43-439c-4cc4-9070-51cd19ca3ff4'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:49.568057 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d0353f43-439c-4cc4-9070-51cd19ca3ff4 admin admin] The token cache is being invalidate because project d39c13d4012549c2890bf1d904d1c041 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:49:49.568408 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d0353f43-439c-4cc4-9070-51cd19ca3ff4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:49.569117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d0353f43-439c-4cc4-9070-51cd19ca3ff4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:49.570685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1008/2016] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:49:49 2022] DELETE /identity/v3/projects/d39c13d4012549c2890bf1d904d1c041 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:49.576308 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1a0bcb-6b8f-4d36-bf05-ff57a57b4f0b None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:49.576792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1a0bcb-6b8f-4d36-bf05-ff57a57b4f0b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:49.576883 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1a0bcb-6b8f-4d36-bf05-ff57a57b4f0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:49.596533 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-af1a0bcb-6b8f-4d36-bf05-ff57a57b4f0b None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:49.597743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-af1a0bcb-6b8f-4d36-bf05-ff57a57b4f0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:49.665941 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1009/2017] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:49.675201 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ac24c6-c1c8-4fe2-94cd-ae00e55393b1 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:49.677002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ac24c6-c1c8-4fe2-94cd-ae00e55393b1 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:49.677002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ac24c6-c1c8-4fe2-94cd-ae00e55393b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:49.699056 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-f5ac24c6-c1c8-4fe2-94cd-ae00e55393b1 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:49.700927 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-f5ac24c6-c1c8-4fe2-94cd-ae00e55393b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:49.708909 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1009/2018] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:49.713200 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1a0bcb-6b8f-4d36-bf05-ff57a57b4f0b None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:49.723836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1a0bcb-6b8f-4d36-bf05-ff57a57b4f0b None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:49.729071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7722dec-f9ab-41b5-a2b2-25caf9cc3ef7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:49.729685 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7722dec-f9ab-41b5-a2b2-25caf9cc3ef7 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:49.730038 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7722dec-f9ab-41b5-a2b2-25caf9cc3ef7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:49.730325 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7722dec-f9ab-41b5-a2b2-25caf9cc3ef7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:49.731030 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7722dec-f9ab-41b5-a2b2-25caf9cc3ef7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:49:49.731423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7722dec-f9ab-41b5-a2b2-25caf9cc3ef7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:49.733319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7722dec-f9ab-41b5-a2b2-25caf9cc3ef7 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:49.746596 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1010/2019] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:49:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:49:49.751144 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ac24c6-c1c8-4fe2-94cd-ae00e55393b1 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:49.760511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ac24c6-c1c8-4fe2-94cd-ae00e55393b1 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:49.766003 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d814307-4458-458c-aef4-de99cf0a2949 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:49.766614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d814307-4458-458c-aef4-de99cf0a2949 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:49.766898 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d814307-4458-458c-aef4-de99cf0a2949 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:49.767213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d814307-4458-458c-aef4-de99cf0a2949 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:49.767848 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d814307-4458-458c-aef4-de99cf0a2949 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:49.769814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d814307-4458-458c-aef4-de99cf0a2949 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:49.774285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:49.904745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1010/2020] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:49:49 2022] POST /identity/v3/projects => generated 375 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:49.909338 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7722dec-f9ab-41b5-a2b2-25caf9cc3ef7 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:49.919669 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7722dec-f9ab-41b5-a2b2-25caf9cc3ef7 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:49.976147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26009f77-09f2-4485-aa9c-6b72a074e4b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:49.976788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26009f77-09f2-4485-aa9c-6b72a074e4b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:49.977065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26009f77-09f2-4485-aa9c-6b72a074e4b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:49.977744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26009f77-09f2-4485-aa9c-6b72a074e4b2 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:49.977962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26009f77-09f2-4485-aa9c-6b72a074e4b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:49.979915 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26009f77-09f2-4485-aa9c-6b72a074e4b2 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:49.985355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:50.079882 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1011/2021] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:49:49 2022] POST /identity/v3/users => generated 288 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:50.087899 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d814307-4458-458c-aef4-de99cf0a2949 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:50.095955 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d814307-4458-458c-aef4-de99cf0a2949 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:50.100650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8aaaea8-f05a-448f-b7e9-69ec352e792e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:50.101262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8aaaea8-f05a-448f-b7e9-69ec352e792e admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:50.101539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8aaaea8-f05a-448f-b7e9-69ec352e792e admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:50.101840 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8aaaea8-f05a-448f-b7e9-69ec352e792e admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:50.102480 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8aaaea8-f05a-448f-b7e9-69ec352e792e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:50.104094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8aaaea8-f05a-448f-b7e9-69ec352e792e admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:50.115026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1011/2022] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:50 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:49:50.119087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26009f77-09f2-4485-aa9c-6b72a074e4b2 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:50.130223 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26009f77-09f2-4485-aa9c-6b72a074e4b2 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:50.135195 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d964275-f078-4821-962e-eb0afe89d46d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:50.135797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d964275-f078-4821-962e-eb0afe89d46d admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:50.136075 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d964275-f078-4821-962e-eb0afe89d46d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:50.136443 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d964275-f078-4821-962e-eb0afe89d46d admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:50.137094 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d964275-f078-4821-962e-eb0afe89d46d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:50.139532 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d964275-f078-4821-962e-eb0afe89d46d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:50.149918 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1012/2023] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:50 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:50.154340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8aaaea8-f05a-448f-b7e9-69ec352e792e admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:50.164657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8aaaea8-f05a-448f-b7e9-69ec352e792e admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:50.169459 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301653a2-4c28-438c-9428-6f189dd3d738 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:50.170107 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301653a2-4c28-438c-9428-6f189dd3d738 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:50.170389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301653a2-4c28-438c-9428-6f189dd3d738 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:50.170665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301653a2-4c28-438c-9428-6f189dd3d738 admin admin] PATH_INFO: `/v3/projects/84c5a63f68f947679e6dfc67cebee374/users/b75c91b3cf4140fe8e32b9c40e2710ca/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:50.187190 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301653a2-4c28-438c-9428-6f189dd3d738 admin admin] RBAC: Authorizing `identity:create_grant(project_id=84c5a63f68f947679e6dfc67cebee374, user_id=b75c91b3cf4140fe8e32b9c40e2710ca, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:50.190146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301653a2-4c28-438c-9428-6f189dd3d738 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:50.279240 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1012/2024] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:50 2022] PUT /identity/v3/projects/84c5a63f68f947679e6dfc67cebee374/users/b75c91b3cf4140fe8e32b9c40e2710ca/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:50.289625 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf8b1b4-19cd-4ede-bbbd-31682392c20e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:50.289915 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf8b1b4-19cd-4ede-bbbd-31682392c20e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:50.290218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf8b1b4-19cd-4ede-bbbd-31682392c20e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:50.320639 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2cf8b1b4-19cd-4ede-bbbd-31682392c20e None None] MFA Rules not processed for user `b75c91b3cf4140fe8e32b9c40e2710ca`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:50.321839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2cf8b1b4-19cd-4ede-bbbd-31682392c20e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:50.390967 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1013/2025] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:50 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:50.403708 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301653a2-4c28-438c-9428-6f189dd3d738 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:50.406753 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-301653a2-4c28-438c-9428-6f189dd3d738 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:50.482145 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301653a2-4c28-438c-9428-6f189dd3d738 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:50.487804 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f1b795-9120-4430-ad08-e43f7b699234 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:50.488406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f1b795-9120-4430-ad08-e43f7b699234 service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:50.488837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f1b795-9120-4430-ad08-e43f7b699234 service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:50.489427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f1b795-9120-4430-ad08-e43f7b699234 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:50.491258 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:50.502858 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f1b795-9120-4430-ad08-e43f7b699234 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:50.505486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f1b795-9120-4430-ad08-e43f7b699234 service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:50.518100 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1013/2026] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:50 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:50.552621 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:50.757147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf8b1b4-19cd-4ede-bbbd-31682392c20e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:50.820582 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf8b1b4-19cd-4ede-bbbd-31682392c20e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:50.825771 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1c9378-9ca8-4703-8e07-85acae456119 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:50.826423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1c9378-9ca8-4703-8e07-85acae456119 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:50.826744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1c9378-9ca8-4703-8e07-85acae456119 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:50.827062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1c9378-9ca8-4703-8e07-85acae456119 admin admin] PATH_INFO: `/v3/users/b75c91b3cf4140fe8e32b9c40e2710ca` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:50.832058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1c9378-9ca8-4703-8e07-85acae456119 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b75c91b3cf4140fe8e32b9c40e2710ca)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:50.832058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1c9378-9ca8-4703-8e07-85acae456119 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:51.075867 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8b1c9378-9ca8-4703-8e07-85acae456119 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b75c91b3cf4140fe8e32b9c40e2710ca'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:51.079805 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:51.091567 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8b1c9378-9ca8-4703-8e07-85acae456119 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b75c91b3cf4140fe8e32b9c40e2710ca'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:51.192980 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8b1c9378-9ca8-4703-8e07-85acae456119 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b75c91b3cf4140fe8e32b9c40e2710ca'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:51.193602 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8b1c9378-9ca8-4703-8e07-85acae456119 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b75c91b3cf4140fe8e32b9c40e2710ca'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:51.201176 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8b1c9378-9ca8-4703-8e07-85acae456119 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b75c91b3cf4140fe8e32b9c40e2710ca'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:51.207072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1014/2027] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:49:50 2022] DELETE /identity/v3/users/b75c91b3cf4140fe8e32b9c40e2710ca => generated 0 bytes in 452 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:51.211683 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d18b05-82de-4bd7-abd5-ffb8094e42da None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:51.211977 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d18b05-82de-4bd7-abd5-ffb8094e42da None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:51.212253 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d18b05-82de-4bd7-abd5-ffb8094e42da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:51.231125 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-59d18b05-82de-4bd7-abd5-ffb8094e42da None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:51.232820 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-59d18b05-82de-4bd7-abd5-ffb8094e42da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:51.291502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1014/2028] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:51.301476 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1c9378-9ca8-4703-8e07-85acae456119 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:51.303150 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8b1c9378-9ca8-4703-8e07-85acae456119 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:51.362069 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1c9378-9ca8-4703-8e07-85acae456119 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:51.365863 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0570b07-ab55-4079-97c7-c563654e921f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:51.366476 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0570b07-ab55-4079-97c7-c563654e921f service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:51.366757 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0570b07-ab55-4079-97c7-c563654e921f service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:51.367078 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0570b07-ab55-4079-97c7-c563654e921f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:51.377628 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0570b07-ab55-4079-97c7-c563654e921f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:51.379486 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0570b07-ab55-4079-97c7-c563654e921f service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:51.390452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1015/2029] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:51.428335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:51.802492 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d18b05-82de-4bd7-abd5-ffb8094e42da None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:51.803967 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-59d18b05-82de-4bd7-abd5-ffb8094e42da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:51.819771 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d18b05-82de-4bd7-abd5-ffb8094e42da None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:51.827396 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4083543-bd55-41e2-b6b7-6533e3ecf617 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:51.828312 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4083543-bd55-41e2-b6b7-6533e3ecf617 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:51.828603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4083543-bd55-41e2-b6b7-6533e3ecf617 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:51.830885 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4083543-bd55-41e2-b6b7-6533e3ecf617 admin admin] PATH_INFO: `/v3/projects/84c5a63f68f947679e6dfc67cebee374` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:51.830885 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4083543-bd55-41e2-b6b7-6533e3ecf617 admin admin] RBAC: Authorizing `identity:delete_project(project_id=84c5a63f68f947679e6dfc67cebee374)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:51.833171 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4083543-bd55-41e2-b6b7-6533e3ecf617 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:51.953600 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-b4083543-bd55-41e2-b6b7-6533e3ecf617 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '84c5a63f68f947679e6dfc67cebee374', 'request_id': 'req-b4083543-bd55-41e2-b6b7-6533e3ecf617'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:52.033435 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-b4083543-bd55-41e2-b6b7-6533e3ecf617 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '84c5a63f68f947679e6dfc67cebee374', 'request_id': 'req-b4083543-bd55-41e2-b6b7-6533e3ecf617'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:52.041116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-b4083543-bd55-41e2-b6b7-6533e3ecf617 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '84c5a63f68f947679e6dfc67cebee374', 'request_id': 'req-b4083543-bd55-41e2-b6b7-6533e3ecf617'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:52.053871 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-b4083543-bd55-41e2-b6b7-6533e3ecf617 admin admin] The token cache is being invalidate because project 84c5a63f68f947679e6dfc67cebee374 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:49:52.054220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-b4083543-bd55-41e2-b6b7-6533e3ecf617 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:52.054867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-b4083543-bd55-41e2-b6b7-6533e3ecf617 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:49:52.056518 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1015/2030] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:49:51 2022] DELETE /identity/v3/projects/84c5a63f68f947679e6dfc67cebee374 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:52.063551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee3fe8d-3228-4c43-88f9-c4de3384b6f7 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:52.063993 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee3fe8d-3228-4c43-88f9-c4de3384b6f7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:52.064687 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee3fe8d-3228-4c43-88f9-c4de3384b6f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:52.084935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9ee3fe8d-3228-4c43-88f9-c4de3384b6f7 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:52.086135 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9ee3fe8d-3228-4c43-88f9-c4de3384b6f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:52.130401 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=91,keystone:INSERT=3,keystone:DELETE=6 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:52.177065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1016/2031] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:52.189117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:52.189750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb6d72a-459c-4347-8059-247fdf25a812 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:52.190041 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb6d72a-459c-4347-8059-247fdf25a812 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:52.190320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb6d72a-459c-4347-8059-247fdf25a812 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:52.212681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-ccb6d72a-459c-4347-8059-247fdf25a812 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:52.213900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ccb6d72a-459c-4347-8059-247fdf25a812 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:52.229007 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1016/2032] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:52.234116 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee3fe8d-3228-4c43-88f9-c4de3384b6f7 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:52.245884 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee3fe8d-3228-4c43-88f9-c4de3384b6f7 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:52.251848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58566f5-5bcc-455c-a7a4-f9f867d5d7b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:52.252479 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58566f5-5bcc-455c-a7a4-f9f867d5d7b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:52.252765 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58566f5-5bcc-455c-a7a4-f9f867d5d7b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:52.253039 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58566f5-5bcc-455c-a7a4-f9f867d5d7b8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:52.253685 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58566f5-5bcc-455c-a7a4-f9f867d5d7b8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:49:52.253988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58566f5-5bcc-455c-a7a4-f9f867d5d7b8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:52.255652 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58566f5-5bcc-455c-a7a4-f9f867d5d7b8 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:52.267956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1017/2033] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:49:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:49:52.272339 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb6d72a-459c-4347-8059-247fdf25a812 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:52.282265 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb6d72a-459c-4347-8059-247fdf25a812 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:52.287038 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ab3780-e421-4970-b305-b61e4570c484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:52.287648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ab3780-e421-4970-b305-b61e4570c484 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:52.287935 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ab3780-e421-4970-b305-b61e4570c484 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:52.288214 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ab3780-e421-4970-b305-b61e4570c484 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:52.288907 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ab3780-e421-4970-b305-b61e4570c484 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:52.291000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ab3780-e421-4970-b305-b61e4570c484 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:52.421270 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1017/2034] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:49:52 2022] POST /identity/v3/projects => generated 379 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:52.422670 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58566f5-5bcc-455c-a7a4-f9f867d5d7b8 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:52.433546 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58566f5-5bcc-455c-a7a4-f9f867d5d7b8 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:52.492586 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df1e7e9-5ab7-4a1f-9680-3c88213d0162 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:52.493199 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df1e7e9-5ab7-4a1f-9680-3c88213d0162 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:52.493480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df1e7e9-5ab7-4a1f-9680-3c88213d0162 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:52.493756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df1e7e9-5ab7-4a1f-9680-3c88213d0162 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:52.494428 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df1e7e9-5ab7-4a1f-9680-3c88213d0162 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:52.496373 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df1e7e9-5ab7-4a1f-9680-3c88213d0162 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:52.573453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1018/2035] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:49:52 2022] POST /identity/v3/users => generated 290 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:52.580802 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ab3780-e421-4970-b305-b61e4570c484 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:52.594746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ab3780-e421-4970-b305-b61e4570c484 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:52.599560 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa54576f-9423-449b-a4a9-e38986c9b053 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:52.600181 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa54576f-9423-449b-a4a9-e38986c9b053 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:52.605992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa54576f-9423-449b-a4a9-e38986c9b053 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:52.605992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa54576f-9423-449b-a4a9-e38986c9b053 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:52.605992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa54576f-9423-449b-a4a9-e38986c9b053 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:52.605992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa54576f-9423-449b-a4a9-e38986c9b053 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:52.614114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1018/2036] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:52 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:52.618028 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df1e7e9-5ab7-4a1f-9680-3c88213d0162 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:52.628246 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df1e7e9-5ab7-4a1f-9680-3c88213d0162 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:52.633085 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301c4719-a009-4204-9f94-55fd31ed8615 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:52.633688 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301c4719-a009-4204-9f94-55fd31ed8615 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:52.635089 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301c4719-a009-4204-9f94-55fd31ed8615 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:52.635089 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301c4719-a009-4204-9f94-55fd31ed8615 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:52.635089 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301c4719-a009-4204-9f94-55fd31ed8615 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:52.636468 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301c4719-a009-4204-9f94-55fd31ed8615 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:52.647185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1019/2037] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:52 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:52.651207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa54576f-9423-449b-a4a9-e38986c9b053 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:52.661671 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa54576f-9423-449b-a4a9-e38986c9b053 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:52.666370 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905cdad7-e790-4e33-a3d0-9f4f14ce66ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:52.667059 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905cdad7-e790-4e33-a3d0-9f4f14ce66ae admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:52.667388 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905cdad7-e790-4e33-a3d0-9f4f14ce66ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:52.667659 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905cdad7-e790-4e33-a3d0-9f4f14ce66ae admin admin] PATH_INFO: `/v3/projects/e9098ac0ccca474fb0e91a5fffbb36a7/users/45c3f87274c14c4d96eab3816e80d238/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:52.686543 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905cdad7-e790-4e33-a3d0-9f4f14ce66ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9098ac0ccca474fb0e91a5fffbb36a7, user_id=45c3f87274c14c4d96eab3816e80d238, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:52.690230 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905cdad7-e790-4e33-a3d0-9f4f14ce66ae admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:52.791905 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1019/2038] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:52 2022] PUT /identity/v3/projects/e9098ac0ccca474fb0e91a5fffbb36a7/users/45c3f87274c14c4d96eab3816e80d238/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:52.803423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa173edc-6a80-484d-9df8-8bb0822639c7 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:52.803709 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa173edc-6a80-484d-9df8-8bb0822639c7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:52.803980 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa173edc-6a80-484d-9df8-8bb0822639c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:52.831857 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-fa173edc-6a80-484d-9df8-8bb0822639c7 None None] MFA Rules not processed for user `45c3f87274c14c4d96eab3816e80d238`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:52.832982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fa173edc-6a80-484d-9df8-8bb0822639c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:52.899641 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1020/2039] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:52 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:49:52.905365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905cdad7-e790-4e33-a3d0-9f4f14ce66ae admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:52.968389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905cdad7-e790-4e33-a3d0-9f4f14ce66ae admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:52.973191 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafe52d5-9605-4caa-a9c3-9f432adb74a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:52.973810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafe52d5-9605-4caa-a9c3-9f432adb74a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:52.974091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafe52d5-9605-4caa-a9c3-9f432adb74a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:52.974365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafe52d5-9605-4caa-a9c3-9f432adb74a2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:52.975030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafe52d5-9605-4caa-a9c3-9f432adb74a2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:52.976938 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafe52d5-9605-4caa-a9c3-9f432adb74a2 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:53.120629 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1020/2040] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:49:52 2022] POST /identity/v3/projects => generated 379 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:53.125731 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa173edc-6a80-484d-9df8-8bb0822639c7 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:53.147564 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa173edc-6a80-484d-9df8-8bb0822639c7 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:53.224367 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfdfaa7-0e32-430e-8b25-923cd0b4d698 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:53.225021 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfdfaa7-0e32-430e-8b25-923cd0b4d698 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:53.225313 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfdfaa7-0e32-430e-8b25-923cd0b4d698 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:53.225625 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfdfaa7-0e32-430e-8b25-923cd0b4d698 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:53.226335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfdfaa7-0e32-430e-8b25-923cd0b4d698 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:53.228293 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfdfaa7-0e32-430e-8b25-923cd0b4d698 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:53.321804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1021/2041] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:49:53 2022] POST /identity/v3/users => generated 296 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:49:53.326531 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafe52d5-9605-4caa-a9c3-9f432adb74a2 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:53.336803 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafe52d5-9605-4caa-a9c3-9f432adb74a2 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:53.341644 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe2447c-ba04-4704-b92a-09aebc180145 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:53.342184 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe2447c-ba04-4704-b92a-09aebc180145 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:53.342471 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe2447c-ba04-4704-b92a-09aebc180145 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:53.342740 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe2447c-ba04-4704-b92a-09aebc180145 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:53.343406 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe2447c-ba04-4704-b92a-09aebc180145 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:53.344991 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe2447c-ba04-4704-b92a-09aebc180145 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:53.355481 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1021/2042] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:53 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:53.363024 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfdfaa7-0e32-430e-8b25-923cd0b4d698 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:53.373838 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfdfaa7-0e32-430e-8b25-923cd0b4d698 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:53.378564 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800bdf7c-b461-4885-9551-cd7d33380f60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:53.379257 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800bdf7c-b461-4885-9551-cd7d33380f60 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:53.379544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800bdf7c-b461-4885-9551-cd7d33380f60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:53.379824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800bdf7c-b461-4885-9551-cd7d33380f60 admin admin] PATH_INFO: `/v3/projects/36bccc4bc59d4ff9b7f0d4edb908c818/users/d2a1e56c670842abbf43028dfe082e25/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:53.380272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:53.396359 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800bdf7c-b461-4885-9551-cd7d33380f60 admin admin] RBAC: Authorizing `identity:create_grant(project_id=36bccc4bc59d4ff9b7f0d4edb908c818, user_id=d2a1e56c670842abbf43028dfe082e25, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:53.399257 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800bdf7c-b461-4885-9551-cd7d33380f60 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:53.577380 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1022/2043] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:53 2022] PUT /identity/v3/projects/36bccc4bc59d4ff9b7f0d4edb908c818/users/d2a1e56c670842abbf43028dfe082e25/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:53.581536 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe2447c-ba04-4704-b92a-09aebc180145 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:53.592363 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe2447c-ba04-4704-b92a-09aebc180145 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:53.592917 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=94,keystone:INSERT=3,keystone:DELETE=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:53.651628 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee189e2-82a1-47c8-962b-962a4f11967f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:53.652244 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee189e2-82a1-47c8-962b-962a4f11967f admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:53.652529 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee189e2-82a1-47c8-962b-962a4f11967f admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:53.652807 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee189e2-82a1-47c8-962b-962a4f11967f admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:53.653694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee189e2-82a1-47c8-962b-962a4f11967f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:53.655489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee189e2-82a1-47c8-962b-962a4f11967f admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:53.665231 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1022/2044] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:49:53 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:49:53.670030 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800bdf7c-b461-4885-9551-cd7d33380f60 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:53.680884 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800bdf7c-b461-4885-9551-cd7d33380f60 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:53.686464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fa6234-b31a-404d-8c13-4af85758a762 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:53.687163 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fa6234-b31a-404d-8c13-4af85758a762 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:53.687449 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fa6234-b31a-404d-8c13-4af85758a762 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:53.687726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fa6234-b31a-404d-8c13-4af85758a762 admin admin] PATH_INFO: `/v3/projects/36bccc4bc59d4ff9b7f0d4edb908c818/users/d2a1e56c670842abbf43028dfe082e25/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:53.689720 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98fa6234-b31a-404d-8c13-4af85758a762 admin admin] RBAC: Authorizing `identity:create_grant(project_id=36bccc4bc59d4ff9b7f0d4edb908c818, user_id=d2a1e56c670842abbf43028dfe082e25, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:53.692393 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98fa6234-b31a-404d-8c13-4af85758a762 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:53.702872 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1023/2045] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:49:53 2022] PUT /identity/v3/projects/36bccc4bc59d4ff9b7f0d4edb908c818/users/d2a1e56c670842abbf43028dfe082e25/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:49:53.714368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec3d6a0-31ca-4673-bb9a-9268f3a7175e None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:53.714655 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec3d6a0-31ca-4673-bb9a-9268f3a7175e None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:53.714959 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec3d6a0-31ca-4673-bb9a-9268f3a7175e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:53.743067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-5ec3d6a0-31ca-4673-bb9a-9268f3a7175e None None] MFA Rules not processed for user `d2a1e56c670842abbf43028dfe082e25`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:49:53.744310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-5ec3d6a0-31ca-4673-bb9a-9268f3a7175e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:53.824919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1023/2046] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:49:53 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:49:53.838305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fa6234-b31a-404d-8c13-4af85758a762 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:53.840053 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-98fa6234-b31a-404d-8c13-4af85758a762 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:49:53.900946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fa6234-b31a-404d-8c13-4af85758a762 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:53.905432 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18be0a14-f55d-4b37-9051-089859ef6e1b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:53.906058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18be0a14-f55d-4b37-9051-089859ef6e1b service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:53.906342 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18be0a14-f55d-4b37-9051-089859ef6e1b service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:53.906618 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18be0a14-f55d-4b37-9051-089859ef6e1b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:53.916215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18be0a14-f55d-4b37-9051-089859ef6e1b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:53.917641 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18be0a14-f55d-4b37-9051-089859ef6e1b service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:53.928207 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1024/2047] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:53 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:54.133473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec3d6a0-31ca-4673-bb9a-9268f3a7175e None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:49:54.143303 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec3d6a0-31ca-4673-bb9a-9268f3a7175e None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:49:54.146836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bac9306-7cf1-4581-8c01-9ccdd33bf6e1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:49:54.148753 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bac9306-7cf1-4581-8c01-9ccdd33bf6e1 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:49:54.148753 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bac9306-7cf1-4581-8c01-9ccdd33bf6e1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:49:54.148753 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bac9306-7cf1-4581-8c01-9ccdd33bf6e1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:49:54.205183 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bac9306-7cf1-4581-8c01-9ccdd33bf6e1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:49:54.206624 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bac9306-7cf1-4581-8c01-9ccdd33bf6e1 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:49:54.215635 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1024/2048] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:49:54 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:49:55.604749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:55.724825 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:56.182300 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:56.576243 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:57.330860 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:58.382099 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:49:59.483065 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:00.284812 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:10.495851 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=33 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:10.632890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=28 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:11.435686 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18be0a14-f55d-4b37-9051-089859ef6e1b service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:11.504008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18be0a14-f55d-4b37-9051-089859ef6e1b service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:11.508829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4932183e-68b2-4886-b2b4-dffa2fdc18fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:11.509599 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4932183e-68b2-4886-b2b4-dffa2fdc18fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:11.509987 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4932183e-68b2-4886-b2b4-dffa2fdc18fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:11.510369 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4932183e-68b2-4886-b2b4-dffa2fdc18fc admin admin] PATH_INFO: `/v3/users/45c3f87274c14c4d96eab3816e80d238` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:11.511685 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4932183e-68b2-4886-b2b4-dffa2fdc18fc admin admin] RBAC: Authorizing `identity:delete_user(user_id=45c3f87274c14c4d96eab3816e80d238)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:11.514841 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4932183e-68b2-4886-b2b4-dffa2fdc18fc admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:11.744180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4932183e-68b2-4886-b2b4-dffa2fdc18fc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '45c3f87274c14c4d96eab3816e80d238'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:11.756822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4932183e-68b2-4886-b2b4-dffa2fdc18fc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '45c3f87274c14c4d96eab3816e80d238'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:11.873806 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4932183e-68b2-4886-b2b4-dffa2fdc18fc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '45c3f87274c14c4d96eab3816e80d238'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:11.874603 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4932183e-68b2-4886-b2b4-dffa2fdc18fc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '45c3f87274c14c4d96eab3816e80d238'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:11.883168 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4932183e-68b2-4886-b2b4-dffa2fdc18fc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '45c3f87274c14c4d96eab3816e80d238'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:11.889834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1025/2049] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:50:11 2022] DELETE /identity/v3/users/45c3f87274c14c4d96eab3816e80d238 => generated 0 bytes in 455 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:50:11.894365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bc87bb-daf0-42dc-9e3a-7a603886d947 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:11.894866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bc87bb-daf0-42dc-9e3a-7a603886d947 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:11.894997 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bc87bb-daf0-42dc-9e3a-7a603886d947 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:11.914055 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-49bc87bb-daf0-42dc-9e3a-7a603886d947 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:11.915303 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-49bc87bb-daf0-42dc-9e3a-7a603886d947 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:11.982262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1025/2050] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:50:11.994128 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4932183e-68b2-4886-b2b4-dffa2fdc18fc admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:11.995624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4932183e-68b2-4886-b2b4-dffa2fdc18fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:12.057436 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4932183e-68b2-4886-b2b4-dffa2fdc18fc admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:12.061753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f038eadf-6a7a-451e-8f3c-0f058de3abd3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:12.062357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f038eadf-6a7a-451e-8f3c-0f058de3abd3 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:12.062630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f038eadf-6a7a-451e-8f3c-0f058de3abd3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:12.062897 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f038eadf-6a7a-451e-8f3c-0f058de3abd3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:12.071655 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f038eadf-6a7a-451e-8f3c-0f058de3abd3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:12.073153 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f038eadf-6a7a-451e-8f3c-0f058de3abd3 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:12.083300 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1026/2051] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:50:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:50:12.498636 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49bc87bb-daf0-42dc-9e3a-7a603886d947 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:12.498636 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-49bc87bb-daf0-42dc-9e3a-7a603886d947 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:12.500858 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49bc87bb-daf0-42dc-9e3a-7a603886d947 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:12.505552 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb393fd9-b9d5-41ad-a771-2b6e2b571d18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:12.506197 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb393fd9-b9d5-41ad-a771-2b6e2b571d18 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:12.506476 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb393fd9-b9d5-41ad-a771-2b6e2b571d18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:12.506747 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb393fd9-b9d5-41ad-a771-2b6e2b571d18 admin admin] PATH_INFO: `/v3/projects/e9098ac0ccca474fb0e91a5fffbb36a7` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:12.507935 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb393fd9-b9d5-41ad-a771-2b6e2b571d18 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e9098ac0ccca474fb0e91a5fffbb36a7)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:12.509941 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb393fd9-b9d5-41ad-a771-2b6e2b571d18 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:12.638626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-eb393fd9-b9d5-41ad-a771-2b6e2b571d18 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e9098ac0ccca474fb0e91a5fffbb36a7', 'request_id': 'req-eb393fd9-b9d5-41ad-a771-2b6e2b571d18'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:12.724414 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-eb393fd9-b9d5-41ad-a771-2b6e2b571d18 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e9098ac0ccca474fb0e91a5fffbb36a7', 'request_id': 'req-eb393fd9-b9d5-41ad-a771-2b6e2b571d18'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:12.730580 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-eb393fd9-b9d5-41ad-a771-2b6e2b571d18 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e9098ac0ccca474fb0e91a5fffbb36a7', 'request_id': 'req-eb393fd9-b9d5-41ad-a771-2b6e2b571d18'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:12.743552 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-eb393fd9-b9d5-41ad-a771-2b6e2b571d18 admin admin] The token cache is being invalidate because project e9098ac0ccca474fb0e91a5fffbb36a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:50:12.743892 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-eb393fd9-b9d5-41ad-a771-2b6e2b571d18 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:12.744498 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-eb393fd9-b9d5-41ad-a771-2b6e2b571d18 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:12.746670 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1026/2052] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:50:12 2022] DELETE /identity/v3/projects/e9098ac0ccca474fb0e91a5fffbb36a7 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:50:12.750244 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f038eadf-6a7a-451e-8f3c-0f058de3abd3 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:12.760103 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f038eadf-6a7a-451e-8f3c-0f058de3abd3 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:12.780368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:12.821873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97dd217-0c39-4777-969c-de6d855be32d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:12.827367 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97dd217-0c39-4777-969c-de6d855be32d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:12.827367 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97dd217-0c39-4777-969c-de6d855be32d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:12.827367 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97dd217-0c39-4777-969c-de6d855be32d admin admin] PATH_INFO: `/v3/users/d2a1e56c670842abbf43028dfe082e25` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:12.827367 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97dd217-0c39-4777-969c-de6d855be32d admin admin] RBAC: Authorizing `identity:delete_user(user_id=d2a1e56c670842abbf43028dfe082e25)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:12.827367 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97dd217-0c39-4777-969c-de6d855be32d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:12.872704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb393fd9-b9d5-41ad-a771-2b6e2b571d18 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:12.874146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-eb393fd9-b9d5-41ad-a771-2b6e2b571d18 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:12.887267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb393fd9-b9d5-41ad-a771-2b6e2b571d18 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:12.892212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762e4fef-7e83-4c35-935d-e1a16517354b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:12.892852 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762e4fef-7e83-4c35-935d-e1a16517354b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:12.893134 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762e4fef-7e83-4c35-935d-e1a16517354b admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:12.893412 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762e4fef-7e83-4c35-935d-e1a16517354b admin admin] PATH_INFO: `/v3/users/3deb8742f42c4569925fcf4a9eb50405` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:12.894561 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762e4fef-7e83-4c35-935d-e1a16517354b admin admin] RBAC: Authorizing `identity:delete_user(user_id=3deb8742f42c4569925fcf4a9eb50405)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:12.896538 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762e4fef-7e83-4c35-935d-e1a16517354b admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:13.076970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b97dd217-0c39-4777-969c-de6d855be32d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd2a1e56c670842abbf43028dfe082e25'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:13.088457 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b97dd217-0c39-4777-969c-de6d855be32d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd2a1e56c670842abbf43028dfe082e25'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:13.174695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-762e4fef-7e83-4c35-935d-e1a16517354b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3deb8742f42c4569925fcf4a9eb50405'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:13.185610 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b97dd217-0c39-4777-969c-de6d855be32d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd2a1e56c670842abbf43028dfe082e25'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:13.186230 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b97dd217-0c39-4777-969c-de6d855be32d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd2a1e56c670842abbf43028dfe082e25'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:13.193544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b97dd217-0c39-4777-969c-de6d855be32d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd2a1e56c670842abbf43028dfe082e25'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:13.200417 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1027/2053] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:50:12 2022] DELETE /identity/v3/users/d2a1e56c670842abbf43028dfe082e25 => generated 0 bytes in 451 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:50:13.244030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-762e4fef-7e83-4c35-935d-e1a16517354b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3deb8742f42c4569925fcf4a9eb50405'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:13.257492 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-762e4fef-7e83-4c35-935d-e1a16517354b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3deb8742f42c4569925fcf4a9eb50405'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:13.258801 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-762e4fef-7e83-4c35-935d-e1a16517354b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3deb8742f42c4569925fcf4a9eb50405'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:13.267119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:13.269385 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-762e4fef-7e83-4c35-935d-e1a16517354b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3deb8742f42c4569925fcf4a9eb50405'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:13.276365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1027/2054] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:50:12 2022] DELETE /identity/v3/users/3deb8742f42c4569925fcf4a9eb50405 => generated 0 bytes in 404 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:50:13.282621 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b835384e-9275-499a-a204-4b08c7e42cc5 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:13.283083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b835384e-9275-499a-a204-4b08c7e42cc5 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:13.284328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b835384e-9275-499a-a204-4b08c7e42cc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:13.309513 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-b835384e-9275-499a-a204-4b08c7e42cc5 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:13.311115 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b835384e-9275-499a-a204-4b08c7e42cc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:13.378732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1028/2055] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:50:13.396472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762e4fef-7e83-4c35-935d-e1a16517354b admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:13.398078 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-762e4fef-7e83-4c35-935d-e1a16517354b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:13.478280 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762e4fef-7e83-4c35-935d-e1a16517354b admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:13.484642 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c5e5de-d8fd-4fa7-b31e-2c79295067bd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:13.485405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c5e5de-d8fd-4fa7-b31e-2c79295067bd service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:13.485812 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c5e5de-d8fd-4fa7-b31e-2c79295067bd service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:13.486099 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c5e5de-d8fd-4fa7-b31e-2c79295067bd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:13.499666 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c5e5de-d8fd-4fa7-b31e-2c79295067bd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:13.501388 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c5e5de-d8fd-4fa7-b31e-2c79295067bd service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:13.514213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1028/2056] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:50:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:50:13.854997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b835384e-9275-499a-a204-4b08c7e42cc5 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:13.854997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b835384e-9275-499a-a204-4b08c7e42cc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:13.863541 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b835384e-9275-499a-a204-4b08c7e42cc5 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:13.868997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be10a0c-4114-494a-9b52-fdf796ae9050 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:13.869648 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be10a0c-4114-494a-9b52-fdf796ae9050 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:13.869929 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be10a0c-4114-494a-9b52-fdf796ae9050 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:13.870203 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be10a0c-4114-494a-9b52-fdf796ae9050 admin admin] PATH_INFO: `/v3/projects/36bccc4bc59d4ff9b7f0d4edb908c818` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:13.871593 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be10a0c-4114-494a-9b52-fdf796ae9050 admin admin] RBAC: Authorizing `identity:delete_project(project_id=36bccc4bc59d4ff9b7f0d4edb908c818)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:13.874138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be10a0c-4114-494a-9b52-fdf796ae9050 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:14.052319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:14.062515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5be10a0c-4114-494a-9b52-fdf796ae9050 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '36bccc4bc59d4ff9b7f0d4edb908c818', 'request_id': 'req-5be10a0c-4114-494a-9b52-fdf796ae9050'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:14.070357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5be10a0c-4114-494a-9b52-fdf796ae9050 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '36bccc4bc59d4ff9b7f0d4edb908c818', 'request_id': 'req-5be10a0c-4114-494a-9b52-fdf796ae9050'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:14.167992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5be10a0c-4114-494a-9b52-fdf796ae9050 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '36bccc4bc59d4ff9b7f0d4edb908c818', 'request_id': 'req-5be10a0c-4114-494a-9b52-fdf796ae9050'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:14.181431 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5be10a0c-4114-494a-9b52-fdf796ae9050 admin admin] The token cache is being invalidate because project 36bccc4bc59d4ff9b7f0d4edb908c818 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:50:14.181848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5be10a0c-4114-494a-9b52-fdf796ae9050 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:14.182662 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5be10a0c-4114-494a-9b52-fdf796ae9050 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:14.185010 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1029/2057] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:50:13 2022] DELETE /identity/v3/projects/36bccc4bc59d4ff9b7f0d4edb908c818 => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:50:14.189898 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ad7e4f-b8a4-4563-9059-1511a8dbb026 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:14.190371 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ad7e4f-b8a4-4563-9059-1511a8dbb026 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:14.190451 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ad7e4f-b8a4-4563-9059-1511a8dbb026 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:14.211558 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-f7ad7e4f-b8a4-4563-9059-1511a8dbb026 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:14.212792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-f7ad7e4f-b8a4-4563-9059-1511a8dbb026 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:14.239399 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be10a0c-4114-494a-9b52-fdf796ae9050 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:14.243880 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5be10a0c-4114-494a-9b52-fdf796ae9050 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:14.334946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1029/2058] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:50:14.340914 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc3f0a3-dd57-4d92-ae8c-950a28944977 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:14.347711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc3f0a3-dd57-4d92-ae8c-950a28944977 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:14.347711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc3f0a3-dd57-4d92-ae8c-950a28944977 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:14.347940 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be10a0c-4114-494a-9b52-fdf796ae9050 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:14.347940 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60abbe0-add8-472d-8497-cf840dd94ed8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:14.348258 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60abbe0-add8-472d-8497-cf840dd94ed8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:14.348539 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60abbe0-add8-472d-8497-cf840dd94ed8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:14.348824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60abbe0-add8-472d-8497-cf840dd94ed8 admin admin] PATH_INFO: `/v3/projects/f71250c81e7c47d7ae28db633a23d6de` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:14.349967 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60abbe0-add8-472d-8497-cf840dd94ed8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f71250c81e7c47d7ae28db633a23d6de)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:14.351944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60abbe0-add8-472d-8497-cf840dd94ed8 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:14.361692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-1cc3f0a3-dd57-4d92-ae8c-950a28944977 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:14.362819 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1cc3f0a3-dd57-4d92-ae8c-950a28944977 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:14.370895 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1030/2059] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:50:14.375168 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc3f0a3-dd57-4d92-ae8c-950a28944977 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:14.384484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc3f0a3-dd57-4d92-ae8c-950a28944977 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:14.389248 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f103c47-7994-40b0-b26c-932f7e3c2522 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:14.389844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f103c47-7994-40b0-b26c-932f7e3c2522 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:14.390127 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f103c47-7994-40b0-b26c-932f7e3c2522 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:14.390399 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f103c47-7994-40b0-b26c-932f7e3c2522 admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:14.391074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f103c47-7994-40b0-b26c-932f7e3c2522 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:50:14.391380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f103c47-7994-40b0-b26c-932f7e3c2522 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:14.392958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f103c47-7994-40b0-b26c-932f7e3c2522 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:14.405163 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1031/2060] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:50:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:50:14.409510 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f103c47-7994-40b0-b26c-932f7e3c2522 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:14.419518 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f103c47-7994-40b0-b26c-932f7e3c2522 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:14.424087 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8dcb46c-ceaf-45b7-8bf1-02b778b3dab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:14.424688 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dcb46c-ceaf-45b7-8bf1-02b778b3dab3 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:14.424941 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dcb46c-ceaf-45b7-8bf1-02b778b3dab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:14.425217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dcb46c-ceaf-45b7-8bf1-02b778b3dab3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:14.425830 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8dcb46c-ceaf-45b7-8bf1-02b778b3dab3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:14.427985 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8dcb46c-ceaf-45b7-8bf1-02b778b3dab3 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:14.490528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-e60abbe0-add8-472d-8497-cf840dd94ed8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f71250c81e7c47d7ae28db633a23d6de', 'request_id': 'req-e60abbe0-add8-472d-8497-cf840dd94ed8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:14.497207 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-e60abbe0-add8-472d-8497-cf840dd94ed8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f71250c81e7c47d7ae28db633a23d6de', 'request_id': 'req-e60abbe0-add8-472d-8497-cf840dd94ed8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:14.541368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1032/2061] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:50:14 2022] POST /identity/v3/projects => generated 369 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:50:14.545315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8dcb46c-ceaf-45b7-8bf1-02b778b3dab3 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:14.563601 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-e60abbe0-add8-472d-8497-cf840dd94ed8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f71250c81e7c47d7ae28db633a23d6de', 'request_id': 'req-e60abbe0-add8-472d-8497-cf840dd94ed8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:14.585653 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-e60abbe0-add8-472d-8497-cf840dd94ed8 admin admin] The token cache is being invalidate because project f71250c81e7c47d7ae28db633a23d6de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:50:14.586045 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-e60abbe0-add8-472d-8497-cf840dd94ed8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:14.587066 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-e60abbe0-add8-472d-8497-cf840dd94ed8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:14.589495 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1030/2062] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:50:14 2022] DELETE /identity/v3/projects/f71250c81e7c47d7ae28db633a23d6de => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:50:14.595171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64c64a6-c9f1-4532-ae25-f2b465525c26 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:14.595463 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64c64a6-c9f1-4532-ae25-f2b465525c26 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:14.600056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64c64a6-c9f1-4532-ae25-f2b465525c26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:14.615474 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:14.616644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-c64c64a6-c9f1-4532-ae25-f2b465525c26 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:14.617843 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c64c64a6-c9f1-4532-ae25-f2b465525c26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:14.618971 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8dcb46c-ceaf-45b7-8bf1-02b778b3dab3 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:14.623791 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18031663-629b-474b-9b27-7cf05c6807d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:14.624461 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18031663-629b-474b-9b27-7cf05c6807d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:14.624738 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18031663-629b-474b-9b27-7cf05c6807d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:14.630705 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18031663-629b-474b-9b27-7cf05c6807d9 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:14.630705 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18031663-629b-474b-9b27-7cf05c6807d9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:14.630871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1031/2063] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:50:14.630970 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18031663-629b-474b-9b27-7cf05c6807d9 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:14.633460 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9c8563-5b75-48b4-8750-f0ffec064786 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:14.633899 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9c8563-5b75-48b4-8750-f0ffec064786 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:14.634182 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9c8563-5b75-48b4-8750-f0ffec064786 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:14.655641 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ca9c8563-5b75-48b4-8750-f0ffec064786 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:14.657595 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ca9c8563-5b75-48b4-8750-f0ffec064786 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:14.665917 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1032/2064] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:50:14.670205 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9c8563-5b75-48b4-8750-f0ffec064786 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:14.679635 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9c8563-5b75-48b4-8750-f0ffec064786 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:14.684630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962f03d1-f0b5-4a6e-ae6a-0245afb7864b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:14.685229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962f03d1-f0b5-4a6e-ae6a-0245afb7864b admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:14.685514 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962f03d1-f0b5-4a6e-ae6a-0245afb7864b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:14.685792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962f03d1-f0b5-4a6e-ae6a-0245afb7864b admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:14.686453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962f03d1-f0b5-4a6e-ae6a-0245afb7864b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:50:14.686759 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962f03d1-f0b5-4a6e-ae6a-0245afb7864b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:14.688534 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962f03d1-f0b5-4a6e-ae6a-0245afb7864b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:14.700790 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1033/2065] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:50:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:50:14.705493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962f03d1-f0b5-4a6e-ae6a-0245afb7864b admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:14.716303 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962f03d1-f0b5-4a6e-ae6a-0245afb7864b admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:14.721456 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8beef03-7def-47c5-91c9-e2ee98d22694 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:14.721951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8beef03-7def-47c5-91c9-e2ee98d22694 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:14.722185 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1033/2066] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:50:14 2022] POST /identity/v3/users => generated 285 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:50:14.722406 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8beef03-7def-47c5-91c9-e2ee98d22694 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:14.722694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8beef03-7def-47c5-91c9-e2ee98d22694 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:14.723362 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8beef03-7def-47c5-91c9-e2ee98d22694 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:14.725351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18031663-629b-474b-9b27-7cf05c6807d9 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:14.725481 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8beef03-7def-47c5-91c9-e2ee98d22694 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:14.727147 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-18031663-629b-474b-9b27-7cf05c6807d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:14.754245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18031663-629b-474b-9b27-7cf05c6807d9 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:14.761774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1ebfc4-bf04-4713-a2f2-e532ea3638e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:14.763055 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1ebfc4-bf04-4713-a2f2-e532ea3638e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:14.763339 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1ebfc4-bf04-4713-a2f2-e532ea3638e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:14.763615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1ebfc4-bf04-4713-a2f2-e532ea3638e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:14.764271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1ebfc4-bf04-4713-a2f2-e532ea3638e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:14.765919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1ebfc4-bf04-4713-a2f2-e532ea3638e6 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:14.776508 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1034/2067] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:50:14 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:50:14.781105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1ebfc4-bf04-4713-a2f2-e532ea3638e6 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:14.790830 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1ebfc4-bf04-4713-a2f2-e532ea3638e6 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:14.795807 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc5caec-eea1-4af2-aa16-e3113d2ef77a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:14.796457 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc5caec-eea1-4af2-aa16-e3113d2ef77a admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:14.796743 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc5caec-eea1-4af2-aa16-e3113d2ef77a admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:14.797043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc5caec-eea1-4af2-aa16-e3113d2ef77a admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:14.797833 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc5caec-eea1-4af2-aa16-e3113d2ef77a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:14.799473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc5caec-eea1-4af2-aa16-e3113d2ef77a admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:14.811089 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1035/2068] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:50:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:50:14.815191 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc5caec-eea1-4af2-aa16-e3113d2ef77a admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:14.826624 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc5caec-eea1-4af2-aa16-e3113d2ef77a admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:14.831694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23fbb44-513b-49dc-87e1-5919499fe0bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:14.832202 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23fbb44-513b-49dc-87e1-5919499fe0bd admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:14.833113 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23fbb44-513b-49dc-87e1-5919499fe0bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:14.833420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23fbb44-513b-49dc-87e1-5919499fe0bd admin admin] PATH_INFO: `/v3/projects/c4a51bd6e3f645ddadc8cc69f514add6/users/fe3ebb087df34e09894398edfde53fd8/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:14.840938 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1034/2069] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:50:14 2022] POST /identity/v3/projects => generated 389 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:50:14.842082 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8beef03-7def-47c5-91c9-e2ee98d22694 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:14.851557 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23fbb44-513b-49dc-87e1-5919499fe0bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4a51bd6e3f645ddadc8cc69f514add6, user_id=fe3ebb087df34e09894398edfde53fd8, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:14.854418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23fbb44-513b-49dc-87e1-5919499fe0bd admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:14.933228 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8beef03-7def-47c5-91c9-e2ee98d22694 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:14.940227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85e171a-4fb1-4a12-a3ab-d4f73239278c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:14.940859 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85e171a-4fb1-4a12-a3ab-d4f73239278c admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:14.941191 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85e171a-4fb1-4a12-a3ab-d4f73239278c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:14.941628 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85e171a-4fb1-4a12-a3ab-d4f73239278c admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:14.942271 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85e171a-4fb1-4a12-a3ab-d4f73239278c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:14.944932 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85e171a-4fb1-4a12-a3ab-d4f73239278c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:14.960081 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1036/2070] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:50:14 2022] PUT /identity/v3/projects/c4a51bd6e3f645ddadc8cc69f514add6/users/fe3ebb087df34e09894398edfde53fd8/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:50:14.975958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125b2e27-9ac9-48e3-b851-b1d430ec1ba2 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:14.976240 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125b2e27-9ac9-48e3-b851-b1d430ec1ba2 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:14.976519 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125b2e27-9ac9-48e3-b851-b1d430ec1ba2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:15.018872 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-125b2e27-9ac9-48e3-b851-b1d430ec1ba2 None None] MFA Rules not processed for user `fe3ebb087df34e09894398edfde53fd8`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:15.018872 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-125b2e27-9ac9-48e3-b851-b1d430ec1ba2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:15.046952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1035/2071] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:50:14 2022] POST /identity/v3/users => generated 295 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:50:15.052405 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85e171a-4fb1-4a12-a3ab-d4f73239278c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:15.066465 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85e171a-4fb1-4a12-a3ab-d4f73239278c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:15.112354 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1037/2072] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:14 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:50:15.117226 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125b2e27-9ac9-48e3-b851-b1d430ec1ba2 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:15.150006 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4285e840-4f76-4620-a0c4-d756623ee638 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:15.150625 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4285e840-4f76-4620-a0c4-d756623ee638 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:15.150930 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4285e840-4f76-4620-a0c4-d756623ee638 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:15.151203 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4285e840-4f76-4620-a0c4-d756623ee638 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:15.151925 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4285e840-4f76-4620-a0c4-d756623ee638 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:15.154421 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4285e840-4f76-4620-a0c4-d756623ee638 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:15.165904 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1036/2073] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:50:15 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:50:15.170646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4285e840-4f76-4620-a0c4-d756623ee638 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:15.180459 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4285e840-4f76-4620-a0c4-d756623ee638 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:15.184663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125b2e27-9ac9-48e3-b851-b1d430ec1ba2 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:15.185193 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c12b9f8-5898-40de-98ea-fae6545e2bab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:15.185771 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c12b9f8-5898-40de-98ea-fae6545e2bab admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:15.186093 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c12b9f8-5898-40de-98ea-fae6545e2bab admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:15.186366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c12b9f8-5898-40de-98ea-fae6545e2bab admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:15.187035 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c12b9f8-5898-40de-98ea-fae6545e2bab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:15.188659 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c12b9f8-5898-40de-98ea-fae6545e2bab admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:15.189754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e88df5-7b2c-4261-848d-ce426e397d1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:15.198026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e88df5-7b2c-4261-848d-ce426e397d1b admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:15.198026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e88df5-7b2c-4261-848d-ce426e397d1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:15.198026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e88df5-7b2c-4261-848d-ce426e397d1b admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:15.198026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e88df5-7b2c-4261-848d-ce426e397d1b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:15.198026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e88df5-7b2c-4261-848d-ce426e397d1b admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:15.199224 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1037/2074] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:50:15 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:50:15.203877 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c12b9f8-5898-40de-98ea-fae6545e2bab admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:15.213162 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c12b9f8-5898-40de-98ea-fae6545e2bab admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:15.218535 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fccdf4-9445-499d-b1d5-4390ce76ff7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:15.220053 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fccdf4-9445-499d-b1d5-4390ce76ff7c admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:15.220053 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fccdf4-9445-499d-b1d5-4390ce76ff7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:15.220053 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fccdf4-9445-499d-b1d5-4390ce76ff7c admin admin] PATH_INFO: `/v3/projects/83260468ba544de7b7101469765fe9e5/users/1004952b65844462b8c500300c32d55b/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:15.234691 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35fccdf4-9445-499d-b1d5-4390ce76ff7c admin admin] RBAC: Authorizing `identity:create_grant(project_id=83260468ba544de7b7101469765fe9e5, user_id=1004952b65844462b8c500300c32d55b, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:15.237415 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35fccdf4-9445-499d-b1d5-4390ce76ff7c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:15.329384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1038/2075] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:50:15 2022] POST /identity/v3/projects => generated 369 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:50:15.330615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e88df5-7b2c-4261-848d-ce426e397d1b admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:15.375690 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1038/2076] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:50:15 2022] PUT /identity/v3/projects/83260468ba544de7b7101469765fe9e5/users/1004952b65844462b8c500300c32d55b/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:50:15.386967 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d941c51-2e3e-411a-ad2c-d91b3604cd94 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:15.387706 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d941c51-2e3e-411a-ad2c-d91b3604cd94 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:15.388543 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d941c51-2e3e-411a-ad2c-d91b3604cd94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:15.393755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e88df5-7b2c-4261-848d-ce426e397d1b admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:15.419157 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9d941c51-2e3e-411a-ad2c-d91b3604cd94 None None] MFA Rules not processed for user `1004952b65844462b8c500300c32d55b`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:15.420618 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9d941c51-2e3e-411a-ad2c-d91b3604cd94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:15.454535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47351ca-9ef7-4754-ac07-c210c3fdde82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:15.455175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47351ca-9ef7-4754-ac07-c210c3fdde82 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:15.455452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47351ca-9ef7-4754-ac07-c210c3fdde82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:15.455728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47351ca-9ef7-4754-ac07-c210c3fdde82 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:15.456352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47351ca-9ef7-4754-ac07-c210c3fdde82 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:15.458254 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47351ca-9ef7-4754-ac07-c210c3fdde82 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:15.485073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1039/2077] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:15 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:50:15.495851 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d941c51-2e3e-411a-ad2c-d91b3604cd94 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:15.497946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9d941c51-2e3e-411a-ad2c-d91b3604cd94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:15.569650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d941c51-2e3e-411a-ad2c-d91b3604cd94 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:15.573297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13369d32-8b28-4cde-bd58-5f88118efa97 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:15.574046 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13369d32-8b28-4cde-bd58-5f88118efa97 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:15.574181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13369d32-8b28-4cde-bd58-5f88118efa97 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:15.574460 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13369d32-8b28-4cde-bd58-5f88118efa97 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:15.583372 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13369d32-8b28-4cde-bd58-5f88118efa97 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:15.584797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13369d32-8b28-4cde-bd58-5f88118efa97 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:15.593987 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1040/2078] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:50:15 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:50:15.668176 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1039/2079] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:50:15 2022] POST /identity/v3/users => generated 291 bytes in 338 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:50:15.672975 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13369d32-8b28-4cde-bd58-5f88118efa97 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:15.683929 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13369d32-8b28-4cde-bd58-5f88118efa97 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:15.689027 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f0a750-9f71-4aeb-a82b-7d2725f65a67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:15.689773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f0a750-9f71-4aeb-a82b-7d2725f65a67 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:15.690062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f0a750-9f71-4aeb-a82b-7d2725f65a67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:15.690336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f0a750-9f71-4aeb-a82b-7d2725f65a67 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:15.690995 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f0a750-9f71-4aeb-a82b-7d2725f65a67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:15.692590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f0a750-9f71-4aeb-a82b-7d2725f65a67 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:15.701068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:15.711819 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47351ca-9ef7-4754-ac07-c210c3fdde82 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:15.719290 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1041/2080] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:50:15 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:50:15.725339 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47351ca-9ef7-4754-ac07-c210c3fdde82 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:15.731598 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab37132-a57b-45f6-b532-efc54fe86cf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:15.732470 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab37132-a57b-45f6-b532-efc54fe86cf7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:15.734845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab37132-a57b-45f6-b532-efc54fe86cf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:15.734845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab37132-a57b-45f6-b532-efc54fe86cf7 admin admin] PATH_INFO: `/v3/projects/0b5b8a407f284105a1ce9cd7416ea617/users/0b1206081c22470d8570665af8c71392/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:15.749213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab37132-a57b-45f6-b532-efc54fe86cf7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b5b8a407f284105a1ce9cd7416ea617, user_id=0b1206081c22470d8570665af8c71392, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:15.751988 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab37132-a57b-45f6-b532-efc54fe86cf7 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:15.764694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:15.822731 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1040/2081] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:50:15 2022] PUT /identity/v3/projects/0b5b8a407f284105a1ce9cd7416ea617/users/0b1206081c22470d8570665af8c71392/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:50:15.826561 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f0a750-9f71-4aeb-a82b-7d2725f65a67 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:15.836888 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f0a750-9f71-4aeb-a82b-7d2725f65a67 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:15.893589 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb53183-c1c0-4365-a3c7-a949c6621257 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:15.894238 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb53183-c1c0-4365-a3c7-a949c6621257 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:15.894526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb53183-c1c0-4365-a3c7-a949c6621257 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:15.894806 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb53183-c1c0-4365-a3c7-a949c6621257 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:15.895479 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb53183-c1c0-4365-a3c7-a949c6621257 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:15.897111 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb53183-c1c0-4365-a3c7-a949c6621257 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:15.908073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1042/2082] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:50:15 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:50:15.912512 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab37132-a57b-45f6-b532-efc54fe86cf7 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:15.929959 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab37132-a57b-45f6-b532-efc54fe86cf7 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:15.930292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fc09cd-18a1-449e-85a9-cd154c6969c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:15.930980 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fc09cd-18a1-449e-85a9-cd154c6969c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:15.931267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fc09cd-18a1-449e-85a9-cd154c6969c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:15.931546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fc09cd-18a1-449e-85a9-cd154c6969c7 admin admin] PATH_INFO: `/v3/projects/0b5b8a407f284105a1ce9cd7416ea617/users/0b1206081c22470d8570665af8c71392/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:15.933461 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41fc09cd-18a1-449e-85a9-cd154c6969c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b5b8a407f284105a1ce9cd7416ea617, user_id=0b1206081c22470d8570665af8c71392, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:15.936452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41fc09cd-18a1-449e-85a9-cd154c6969c7 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:15.946714 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1041/2083] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:50:15 2022] PUT /identity/v3/projects/0b5b8a407f284105a1ce9cd7416ea617/users/0b1206081c22470d8570665af8c71392/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:50:15.958742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8b4679-975a-4bfd-9a5b-ec5117b2d3d4 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:15.962240 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8b4679-975a-4bfd-9a5b-ec5117b2d3d4 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:15.962240 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8b4679-975a-4bfd-9a5b-ec5117b2d3d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:15.990096 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9b8b4679-975a-4bfd-9a5b-ec5117b2d3d4 None None] MFA Rules not processed for user `0b1206081c22470d8570665af8c71392`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:15.991538 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9b8b4679-975a-4bfd-9a5b-ec5117b2d3d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:16.068509 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1043/2084] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:15 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:50:16.084256 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fc09cd-18a1-449e-85a9-cd154c6969c7 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:16.086132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-41fc09cd-18a1-449e-85a9-cd154c6969c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:16.144208 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fc09cd-18a1-449e-85a9-cd154c6969c7 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:16.147812 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83cac44-a79e-4f82-a92e-d3f2d9aff637 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:16.148444 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83cac44-a79e-4f82-a92e-d3f2d9aff637 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:16.148751 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83cac44-a79e-4f82-a92e-d3f2d9aff637 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:16.149025 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83cac44-a79e-4f82-a92e-d3f2d9aff637 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:16.159886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83cac44-a79e-4f82-a92e-d3f2d9aff637 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:16.161320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83cac44-a79e-4f82-a92e-d3f2d9aff637 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:16.183216 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1042/2085] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:50:16 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:50:16.871501 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:17.187644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:19.656871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8b4679-975a-4bfd-9a5b-ec5117b2d3d4 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:19.728970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8b4679-975a-4bfd-9a5b-ec5117b2d3d4 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:19.736198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e9907e-a875-4a2a-9e13-514b6ae36462 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:19.736962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e9907e-a875-4a2a-9e13-514b6ae36462 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:19.737312 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e9907e-a875-4a2a-9e13-514b6ae36462 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:19.737590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e9907e-a875-4a2a-9e13-514b6ae36462 admin admin] PATH_INFO: `/v3/users/fe3ebb087df34e09894398edfde53fd8` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:19.740256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e9907e-a875-4a2a-9e13-514b6ae36462 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fe3ebb087df34e09894398edfde53fd8)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:19.741488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e9907e-a875-4a2a-9e13-514b6ae36462 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:19.961300 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-44e9907e-a875-4a2a-9e13-514b6ae36462 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fe3ebb087df34e09894398edfde53fd8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:19.976147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-44e9907e-a875-4a2a-9e13-514b6ae36462 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fe3ebb087df34e09894398edfde53fd8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:20.073305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-44e9907e-a875-4a2a-9e13-514b6ae36462 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fe3ebb087df34e09894398edfde53fd8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:20.073982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-44e9907e-a875-4a2a-9e13-514b6ae36462 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fe3ebb087df34e09894398edfde53fd8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:20.082529 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-44e9907e-a875-4a2a-9e13-514b6ae36462 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fe3ebb087df34e09894398edfde53fd8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:20.089163 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1044/2086] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:50:19 2022] DELETE /identity/v3/users/fe3ebb087df34e09894398edfde53fd8 => generated 0 bytes in 433 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:50:20.097538 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75601056-4d60-4616-8b66-d6a759da7952 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:20.098026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75601056-4d60-4616-8b66-d6a759da7952 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:20.098122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75601056-4d60-4616-8b66-d6a759da7952 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:20.118238 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-75601056-4d60-4616-8b66-d6a759da7952 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:20.119469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-75601056-4d60-4616-8b66-d6a759da7952 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:20.131480 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:20.178922 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1043/2087] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:50:20.189397 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e9907e-a875-4a2a-9e13-514b6ae36462 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:20.190937 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-44e9907e-a875-4a2a-9e13-514b6ae36462 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:20.254008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e9907e-a875-4a2a-9e13-514b6ae36462 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:20.258006 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8cdc02-cf06-48c8-ac2d-150e5e68d9bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:20.258722 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8cdc02-cf06-48c8-ac2d-150e5e68d9bf service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:20.259048 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8cdc02-cf06-48c8-ac2d-150e5e68d9bf service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:20.259324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8cdc02-cf06-48c8-ac2d-150e5e68d9bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:20.269907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8cdc02-cf06-48c8-ac2d-150e5e68d9bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:20.271514 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8cdc02-cf06-48c8-ac2d-150e5e68d9bf service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:20.281883 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1045/2088] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:50:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:50:20.787856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75601056-4d60-4616-8b66-d6a759da7952 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:20.789574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-75601056-4d60-4616-8b66-d6a759da7952 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:20.802630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75601056-4d60-4616-8b66-d6a759da7952 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:20.808289 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f2ea17-70fd-4b5a-9d08-92c8c707f842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:20.809903 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f2ea17-70fd-4b5a-9d08-92c8c707f842 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:20.810436 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f2ea17-70fd-4b5a-9d08-92c8c707f842 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:20.810709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f2ea17-70fd-4b5a-9d08-92c8c707f842 admin admin] PATH_INFO: `/v3/projects/c4a51bd6e3f645ddadc8cc69f514add6` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:20.811973 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f2ea17-70fd-4b5a-9d08-92c8c707f842 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c4a51bd6e3f645ddadc8cc69f514add6)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:20.813937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f2ea17-70fd-4b5a-9d08-92c8c707f842 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:20.958147 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-b3f2ea17-70fd-4b5a-9d08-92c8c707f842 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c4a51bd6e3f645ddadc8cc69f514add6', 'request_id': 'req-b3f2ea17-70fd-4b5a-9d08-92c8c707f842'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:21.085545 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-b3f2ea17-70fd-4b5a-9d08-92c8c707f842 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c4a51bd6e3f645ddadc8cc69f514add6', 'request_id': 'req-b3f2ea17-70fd-4b5a-9d08-92c8c707f842'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:21.093287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-b3f2ea17-70fd-4b5a-9d08-92c8c707f842 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c4a51bd6e3f645ddadc8cc69f514add6', 'request_id': 'req-b3f2ea17-70fd-4b5a-9d08-92c8c707f842'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:21.104750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-b3f2ea17-70fd-4b5a-9d08-92c8c707f842 admin admin] The token cache is being invalidate because project c4a51bd6e3f645ddadc8cc69f514add6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:50:21.105428 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-b3f2ea17-70fd-4b5a-9d08-92c8c707f842 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:21.106316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-b3f2ea17-70fd-4b5a-9d08-92c8c707f842 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:21.107978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1044/2089] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:50:20 2022] DELETE /identity/v3/projects/c4a51bd6e3f645ddadc8cc69f514add6 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:50:21.112019 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8cdc02-cf06-48c8-ac2d-150e5e68d9bf service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:21.123061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8cdc02-cf06-48c8-ac2d-150e5e68d9bf service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:21.144339 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:21.185480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89822b3a-c41c-418e-b685-7decfcaa103b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:21.186116 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89822b3a-c41c-418e-b685-7decfcaa103b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:21.186395 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89822b3a-c41c-418e-b685-7decfcaa103b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:21.186666 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89822b3a-c41c-418e-b685-7decfcaa103b admin admin] PATH_INFO: `/v3/users/0b1206081c22470d8570665af8c71392` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:21.187908 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89822b3a-c41c-418e-b685-7decfcaa103b admin admin] RBAC: Authorizing `identity:delete_user(user_id=0b1206081c22470d8570665af8c71392)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:21.189863 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89822b3a-c41c-418e-b685-7decfcaa103b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:21.416586 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-89822b3a-c41c-418e-b685-7decfcaa103b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0b1206081c22470d8570665af8c71392'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:21.430049 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-89822b3a-c41c-418e-b685-7decfcaa103b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0b1206081c22470d8570665af8c71392'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:21.513586 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-89822b3a-c41c-418e-b685-7decfcaa103b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0b1206081c22470d8570665af8c71392'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:21.514297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-89822b3a-c41c-418e-b685-7decfcaa103b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0b1206081c22470d8570665af8c71392'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:21.523027 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-89822b3a-c41c-418e-b685-7decfcaa103b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0b1206081c22470d8570665af8c71392'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:21.531263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1046/2090] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:50:21 2022] DELETE /identity/v3/users/0b1206081c22470d8570665af8c71392 => generated 0 bytes in 420 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:50:21.990443 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f2ea17-70fd-4b5a-9d08-92c8c707f842 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:21.992423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b3f2ea17-70fd-4b5a-9d08-92c8c707f842 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:22.007772 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f2ea17-70fd-4b5a-9d08-92c8c707f842 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:22.079447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba528584-3c17-4f11-bc37-94dc1b1cfb4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:22.080096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba528584-3c17-4f11-bc37-94dc1b1cfb4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:22.080377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba528584-3c17-4f11-bc37-94dc1b1cfb4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:22.080657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba528584-3c17-4f11-bc37-94dc1b1cfb4a admin admin] PATH_INFO: `/v3/projects/0b5b8a407f284105a1ce9cd7416ea617` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:22.081839 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba528584-3c17-4f11-bc37-94dc1b1cfb4a admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b5b8a407f284105a1ce9cd7416ea617)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:22.083816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba528584-3c17-4f11-bc37-94dc1b1cfb4a admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:22.213948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:22.219143 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-ba528584-3c17-4f11-bc37-94dc1b1cfb4a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b5b8a407f284105a1ce9cd7416ea617', 'request_id': 'req-ba528584-3c17-4f11-bc37-94dc1b1cfb4a'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:22.368548 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-ba528584-3c17-4f11-bc37-94dc1b1cfb4a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b5b8a407f284105a1ce9cd7416ea617', 'request_id': 'req-ba528584-3c17-4f11-bc37-94dc1b1cfb4a'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:22.376364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-ba528584-3c17-4f11-bc37-94dc1b1cfb4a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b5b8a407f284105a1ce9cd7416ea617', 'request_id': 'req-ba528584-3c17-4f11-bc37-94dc1b1cfb4a'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:22.388372 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-ba528584-3c17-4f11-bc37-94dc1b1cfb4a admin admin] The token cache is being invalidate because project 0b5b8a407f284105a1ce9cd7416ea617 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:50:22.388659 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-ba528584-3c17-4f11-bc37-94dc1b1cfb4a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:22.389342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-ba528584-3c17-4f11-bc37-94dc1b1cfb4a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:22.390837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1045/2091] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:50:21 2022] DELETE /identity/v3/projects/0b5b8a407f284105a1ce9cd7416ea617 => generated 0 bytes in 402 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:50:22.396014 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b175daec-d97c-48a1-9fa6-299f2c9a98fb None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:22.396500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b175daec-d97c-48a1-9fa6-299f2c9a98fb None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:22.396591 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b175daec-d97c-48a1-9fa6-299f2c9a98fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:22.417119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-b175daec-d97c-48a1-9fa6-299f2c9a98fb None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:22.418476 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b175daec-d97c-48a1-9fa6-299f2c9a98fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:22.514168 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1047/2092] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:50:22.522025 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5680aa3b-9ae1-4831-98b3-019a97406ad3 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:22.522315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5680aa3b-9ae1-4831-98b3-019a97406ad3 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:22.522592 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5680aa3b-9ae1-4831-98b3-019a97406ad3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:22.542944 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-5680aa3b-9ae1-4831-98b3-019a97406ad3 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:22.544855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-5680aa3b-9ae1-4831-98b3-019a97406ad3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:22.553147 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1046/2093] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:50:22.557494 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b175daec-d97c-48a1-9fa6-299f2c9a98fb None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:22.570200 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b175daec-d97c-48a1-9fa6-299f2c9a98fb None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:22.575818 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e291d0-35c3-4135-8fe1-7c0fe22fbd56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:22.576667 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e291d0-35c3-4135-8fe1-7c0fe22fbd56 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:22.576950 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e291d0-35c3-4135-8fe1-7c0fe22fbd56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:22.577240 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e291d0-35c3-4135-8fe1-7c0fe22fbd56 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:22.578260 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e291d0-35c3-4135-8fe1-7c0fe22fbd56 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:50:22.578612 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e291d0-35c3-4135-8fe1-7c0fe22fbd56 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:22.580535 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e291d0-35c3-4135-8fe1-7c0fe22fbd56 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:22.596494 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1048/2094] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:50:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:50:22.600270 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5680aa3b-9ae1-4831-98b3-019a97406ad3 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:22.610094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5680aa3b-9ae1-4831-98b3-019a97406ad3 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:22.615663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77266eb-193a-4111-88a7-3dd10183a59d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:22.616164 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77266eb-193a-4111-88a7-3dd10183a59d admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:22.616462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77266eb-193a-4111-88a7-3dd10183a59d admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:22.616741 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77266eb-193a-4111-88a7-3dd10183a59d admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:22.617379 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77266eb-193a-4111-88a7-3dd10183a59d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:22.619373 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77266eb-193a-4111-88a7-3dd10183a59d admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:22.723854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1047/2095] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:50:22 2022] POST /identity/v3/projects => generated 377 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:50:22.724928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e291d0-35c3-4135-8fe1-7c0fe22fbd56 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:22.735584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e291d0-35c3-4135-8fe1-7c0fe22fbd56 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:22.761908 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:22.797334 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31dd85af-3120-46e0-a3a9-2b70ec6f9900 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:22.797950 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31dd85af-3120-46e0-a3a9-2b70ec6f9900 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:22.798259 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31dd85af-3120-46e0-a3a9-2b70ec6f9900 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:22.798535 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31dd85af-3120-46e0-a3a9-2b70ec6f9900 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:22.799288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31dd85af-3120-46e0-a3a9-2b70ec6f9900 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:22.801417 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31dd85af-3120-46e0-a3a9-2b70ec6f9900 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:22.925664 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1049/2096] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:50:22 2022] POST /identity/v3/users => generated 289 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:50:22.930028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77266eb-193a-4111-88a7-3dd10183a59d admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:22.940993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77266eb-193a-4111-88a7-3dd10183a59d admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:22.945813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9438fbd6-08b9-444b-9cfe-786f5b790809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:22.946413 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9438fbd6-08b9-444b-9cfe-786f5b790809 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:22.946705 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9438fbd6-08b9-444b-9cfe-786f5b790809 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:22.947014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9438fbd6-08b9-444b-9cfe-786f5b790809 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:22.947658 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9438fbd6-08b9-444b-9cfe-786f5b790809 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:22.949294 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9438fbd6-08b9-444b-9cfe-786f5b790809 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:22.960311 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1048/2097] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:50:22 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:50:22.964365 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31dd85af-3120-46e0-a3a9-2b70ec6f9900 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:22.974795 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31dd85af-3120-46e0-a3a9-2b70ec6f9900 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:22.980209 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f43035-0bab-458f-9a7f-393ac4775153 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:22.981336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f43035-0bab-458f-9a7f-393ac4775153 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:22.981612 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f43035-0bab-458f-9a7f-393ac4775153 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:22.981885 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f43035-0bab-458f-9a7f-393ac4775153 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:22.982563 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f43035-0bab-458f-9a7f-393ac4775153 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:22.984256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f43035-0bab-458f-9a7f-393ac4775153 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:22.994173 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1050/2098] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:50:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:50:22.999588 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9438fbd6-08b9-444b-9cfe-786f5b790809 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:23.009433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9438fbd6-08b9-444b-9cfe-786f5b790809 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:23.014314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65fd5e9-dcbc-4a85-93f3-96f0bc20fd22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:23.014996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65fd5e9-dcbc-4a85-93f3-96f0bc20fd22 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:23.015283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65fd5e9-dcbc-4a85-93f3-96f0bc20fd22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:23.015561 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65fd5e9-dcbc-4a85-93f3-96f0bc20fd22 admin admin] PATH_INFO: `/v3/projects/9f34b6812ea54383849ade850525bc27/users/c65b598f95dd45938b010f4eb59959d3/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:23.030840 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65fd5e9-dcbc-4a85-93f3-96f0bc20fd22 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f34b6812ea54383849ade850525bc27, user_id=c65b598f95dd45938b010f4eb59959d3, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:23.033535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65fd5e9-dcbc-4a85-93f3-96f0bc20fd22 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:23.114171 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1049/2099] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:50:22 2022] PUT /identity/v3/projects/9f34b6812ea54383849ade850525bc27/users/c65b598f95dd45938b010f4eb59959d3/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:50:23.126850 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30251f5b-bd96-4345-b2e5-1e795b78d708 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:23.127170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30251f5b-bd96-4345-b2e5-1e795b78d708 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:23.127438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30251f5b-bd96-4345-b2e5-1e795b78d708 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:23.161747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-30251f5b-bd96-4345-b2e5-1e795b78d708 None None] MFA Rules not processed for user `c65b598f95dd45938b010f4eb59959d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:23.163056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-30251f5b-bd96-4345-b2e5-1e795b78d708 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:23.229070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1051/2100] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:23 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:50:23.264303 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65fd5e9-dcbc-4a85-93f3-96f0bc20fd22 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:23.328285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65fd5e9-dcbc-4a85-93f3-96f0bc20fd22 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:23.333154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9aad943-f1e6-4faf-82a9-d168f2bc8a2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:23.333763 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aad943-f1e6-4faf-82a9-d168f2bc8a2d admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:23.334081 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aad943-f1e6-4faf-82a9-d168f2bc8a2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:23.334351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aad943-f1e6-4faf-82a9-d168f2bc8a2d admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:23.335023 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9aad943-f1e6-4faf-82a9-d168f2bc8a2d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:23.336935 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9aad943-f1e6-4faf-82a9-d168f2bc8a2d admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:23.428225 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1050/2101] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:50:23 2022] POST /identity/v3/projects => generated 377 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:50:23.432393 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30251f5b-bd96-4345-b2e5-1e795b78d708 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:23.445215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30251f5b-bd96-4345-b2e5-1e795b78d708 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:23.509647 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5976ebf8-dbfa-42f3-bee2-b70fd32b1648 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:23.510528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976ebf8-dbfa-42f3-bee2-b70fd32b1648 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:23.510875 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976ebf8-dbfa-42f3-bee2-b70fd32b1648 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:23.516610 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976ebf8-dbfa-42f3-bee2-b70fd32b1648 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:23.516610 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5976ebf8-dbfa-42f3-bee2-b70fd32b1648 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:23.516610 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5976ebf8-dbfa-42f3-bee2-b70fd32b1648 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:23.602366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1052/2102] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:50:23 2022] POST /identity/v3/users => generated 295 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:50:23.603645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9aad943-f1e6-4faf-82a9-d168f2bc8a2d admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:23.616580 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9aad943-f1e6-4faf-82a9-d168f2bc8a2d admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:23.621269 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4c3897-494c-463d-90e1-343652748ac2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:23.621924 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4c3897-494c-463d-90e1-343652748ac2 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:23.622203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4c3897-494c-463d-90e1-343652748ac2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:23.622471 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4c3897-494c-463d-90e1-343652748ac2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:23.623150 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4c3897-494c-463d-90e1-343652748ac2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:23.624738 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4c3897-494c-463d-90e1-343652748ac2 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:23.635617 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1051/2103] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:50:23 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:50:23.640331 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5976ebf8-dbfa-42f3-bee2-b70fd32b1648 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:23.651064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5976ebf8-dbfa-42f3-bee2-b70fd32b1648 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:23.657394 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2ea1b1-3b3b-4af2-ab76-be508e24720e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:23.658131 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2ea1b1-3b3b-4af2-ab76-be508e24720e admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:23.658511 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2ea1b1-3b3b-4af2-ab76-be508e24720e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:23.658858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2ea1b1-3b3b-4af2-ab76-be508e24720e admin admin] PATH_INFO: `/v3/projects/56b92073e683464d83fb6182beaef0e1/users/f93a968411b6427abd98aaa68e324c09/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:23.674490 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2ea1b1-3b3b-4af2-ab76-be508e24720e admin admin] RBAC: Authorizing `identity:create_grant(project_id=56b92073e683464d83fb6182beaef0e1, user_id=f93a968411b6427abd98aaa68e324c09, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:23.677193 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2ea1b1-3b3b-4af2-ab76-be508e24720e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:23.774203 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1053/2104] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:50:23 2022] PUT /identity/v3/projects/56b92073e683464d83fb6182beaef0e1/users/f93a968411b6427abd98aaa68e324c09/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:50:23.780012 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4c3897-494c-463d-90e1-343652748ac2 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:23.790382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4c3897-494c-463d-90e1-343652748ac2 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:23.851206 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:23.877323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7662292d-60e5-4ca7-a421-1ee08bca1cc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:23.877902 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7662292d-60e5-4ca7-a421-1ee08bca1cc0 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:23.878255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7662292d-60e5-4ca7-a421-1ee08bca1cc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:23.878600 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7662292d-60e5-4ca7-a421-1ee08bca1cc0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:23.879443 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7662292d-60e5-4ca7-a421-1ee08bca1cc0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:23.881286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7662292d-60e5-4ca7-a421-1ee08bca1cc0 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:23.892631 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1052/2105] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:50:23 2022] GET /identity/v3/roles => generated 1420 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:50:23.896969 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2ea1b1-3b3b-4af2-ab76-be508e24720e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:23.906167 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:23.912270 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2ea1b1-3b3b-4af2-ab76-be508e24720e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:23.918372 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff7248b-d640-4341-b27f-9db759dd0129 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:23.919661 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff7248b-d640-4341-b27f-9db759dd0129 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:23.920255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff7248b-d640-4341-b27f-9db759dd0129 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:23.920663 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff7248b-d640-4341-b27f-9db759dd0129 admin admin] PATH_INFO: `/v3/projects/56b92073e683464d83fb6182beaef0e1/users/f93a968411b6427abd98aaa68e324c09/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:23.922717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff7248b-d640-4341-b27f-9db759dd0129 admin admin] RBAC: Authorizing `identity:create_grant(project_id=56b92073e683464d83fb6182beaef0e1, user_id=f93a968411b6427abd98aaa68e324c09, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:23.926376 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff7248b-d640-4341-b27f-9db759dd0129 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:23.941498 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1054/2106] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:50:23 2022] PUT /identity/v3/projects/56b92073e683464d83fb6182beaef0e1/users/f93a968411b6427abd98aaa68e324c09/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:50:23.953382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e5b06d-b53c-4f72-9470-31183673cbdd None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:23.953680 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e5b06d-b53c-4f72-9470-31183673cbdd None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:23.953952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e5b06d-b53c-4f72-9470-31183673cbdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:23.987647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-15e5b06d-b53c-4f72-9470-31183673cbdd None None] MFA Rules not processed for user `f93a968411b6427abd98aaa68e324c09`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:23.988921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-15e5b06d-b53c-4f72-9470-31183673cbdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:24.075846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1053/2107] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:23 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:50:24.089747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff7248b-d640-4341-b27f-9db759dd0129 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:24.092554 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1ff7248b-d640-4341-b27f-9db759dd0129 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:24.156729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff7248b-d640-4341-b27f-9db759dd0129 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:24.160448 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda48a93-7ce9-4a0d-894e-ab6dc2bc6729 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:24.161060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda48a93-7ce9-4a0d-894e-ab6dc2bc6729 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:24.161336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda48a93-7ce9-4a0d-894e-ab6dc2bc6729 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:24.161606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda48a93-7ce9-4a0d-894e-ab6dc2bc6729 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:24.227826 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda48a93-7ce9-4a0d-894e-ab6dc2bc6729 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:24.229596 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda48a93-7ce9-4a0d-894e-ab6dc2bc6729 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:24.241593 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1055/2108] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:50:24 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:50:33.030254 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e5b06d-b53c-4f72-9470-31183673cbdd None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:33.032892 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-15e5b06d-b53c-4f72-9470-31183673cbdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:33.046811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e5b06d-b53c-4f72-9470-31183673cbdd None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:33.051694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5ccc11-1b35-48ec-bdad-5ad0e689fda2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:33.052555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5ccc11-1b35-48ec-bdad-5ad0e689fda2 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:33.053189 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5ccc11-1b35-48ec-bdad-5ad0e689fda2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:33.053474 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5ccc11-1b35-48ec-bdad-5ad0e689fda2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:33.065401 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5ccc11-1b35-48ec-bdad-5ad0e689fda2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:33.067224 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5ccc11-1b35-48ec-bdad-5ad0e689fda2 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:33.080527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1054/2109] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:50:33 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:50:34.675277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=52,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:42.032236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda48a93-7ce9-4a0d-894e-ab6dc2bc6729 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:42.033789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cda48a93-7ce9-4a0d-894e-ab6dc2bc6729 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:42.112128 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda48a93-7ce9-4a0d-894e-ab6dc2bc6729 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:42.116992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1287c530-1819-4eed-9683-1feb0fb54472 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:42.117680 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1287c530-1819-4eed-9683-1feb0fb54472 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:42.117961 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1287c530-1819-4eed-9683-1feb0fb54472 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:42.118235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1287c530-1819-4eed-9683-1feb0fb54472 admin admin] PATH_INFO: `/v3/users/93af2ba9dd234e6eac7cf25f0be68f9b` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:42.119411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1287c530-1819-4eed-9683-1feb0fb54472 admin admin] RBAC: Authorizing `identity:delete_user(user_id=93af2ba9dd234e6eac7cf25f0be68f9b)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:42.121520 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1287c530-1819-4eed-9683-1feb0fb54472 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:42.347911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1287c530-1819-4eed-9683-1feb0fb54472 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '93af2ba9dd234e6eac7cf25f0be68f9b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:42.361307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1287c530-1819-4eed-9683-1feb0fb54472 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '93af2ba9dd234e6eac7cf25f0be68f9b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:42.444510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1287c530-1819-4eed-9683-1feb0fb54472 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '93af2ba9dd234e6eac7cf25f0be68f9b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:42.445205 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1287c530-1819-4eed-9683-1feb0fb54472 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '93af2ba9dd234e6eac7cf25f0be68f9b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:42.455330 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1287c530-1819-4eed-9683-1feb0fb54472 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '93af2ba9dd234e6eac7cf25f0be68f9b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:42.466942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1056/2110] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:50:42 2022] DELETE /identity/v3/users/93af2ba9dd234e6eac7cf25f0be68f9b => generated 0 bytes in 435 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:50:42.471418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef20eaa8-c0b5-4d40-9cf1-4b3bb5a2ecc6 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:42.472280 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef20eaa8-c0b5-4d40-9cf1-4b3bb5a2ecc6 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:42.472558 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef20eaa8-c0b5-4d40-9cf1-4b3bb5a2ecc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:42.492101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-ef20eaa8-c0b5-4d40-9cf1-4b3bb5a2ecc6 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:42.493426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ef20eaa8-c0b5-4d40-9cf1-4b3bb5a2ecc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:42.554849 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1055/2111] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:50:42.565782 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1287c530-1819-4eed-9683-1feb0fb54472 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:42.567224 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1287c530-1819-4eed-9683-1feb0fb54472 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:42.635673 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1287c530-1819-4eed-9683-1feb0fb54472 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:42.640560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b59b4c4-7213-4cac-9f7a-af09cd90313c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:42.641172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b59b4c4-7213-4cac-9f7a-af09cd90313c service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:42.641549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b59b4c4-7213-4cac-9f7a-af09cd90313c service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:42.641755 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b59b4c4-7213-4cac-9f7a-af09cd90313c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:42.652402 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b59b4c4-7213-4cac-9f7a-af09cd90313c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:42.654110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b59b4c4-7213-4cac-9f7a-af09cd90313c service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:42.670403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1057/2112] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:50:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:50:43.280290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef20eaa8-c0b5-4d40-9cf1-4b3bb5a2ecc6 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:43.281789 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ef20eaa8-c0b5-4d40-9cf1-4b3bb5a2ecc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:43.295535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef20eaa8-c0b5-4d40-9cf1-4b3bb5a2ecc6 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:43.300283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a157b3de-8304-4100-883e-990fd815c6cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:43.300944 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a157b3de-8304-4100-883e-990fd815c6cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:43.301248 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a157b3de-8304-4100-883e-990fd815c6cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:43.301520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a157b3de-8304-4100-883e-990fd815c6cd admin admin] PATH_INFO: `/v3/projects/f55241ec40414308b1b8fc02618a8188` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:43.302661 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a157b3de-8304-4100-883e-990fd815c6cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=f55241ec40414308b1b8fc02618a8188)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:43.304820 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a157b3de-8304-4100-883e-990fd815c6cd admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:43.419638 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:43.422376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a157b3de-8304-4100-883e-990fd815c6cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f55241ec40414308b1b8fc02618a8188', 'request_id': 'req-a157b3de-8304-4100-883e-990fd815c6cd'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:43.495715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a157b3de-8304-4100-883e-990fd815c6cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f55241ec40414308b1b8fc02618a8188', 'request_id': 'req-a157b3de-8304-4100-883e-990fd815c6cd'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:43.502431 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a157b3de-8304-4100-883e-990fd815c6cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f55241ec40414308b1b8fc02618a8188', 'request_id': 'req-a157b3de-8304-4100-883e-990fd815c6cd'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:43.514259 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a157b3de-8304-4100-883e-990fd815c6cd admin admin] The token cache is being invalidate because project f55241ec40414308b1b8fc02618a8188 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:50:43.514572 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a157b3de-8304-4100-883e-990fd815c6cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:43.515237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a157b3de-8304-4100-883e-990fd815c6cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:43.516906 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1056/2113] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:50:43 2022] DELETE /identity/v3/projects/f55241ec40414308b1b8fc02618a8188 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:50:43.522106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc96b53c-43df-4053-8434-17452b4860de None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:43.522595 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc96b53c-43df-4053-8434-17452b4860de None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:43.522689 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc96b53c-43df-4053-8434-17452b4860de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:43.544780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-dc96b53c-43df-4053-8434-17452b4860de None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:43.546170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-dc96b53c-43df-4053-8434-17452b4860de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:43.564929 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:43.609110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1058/2114] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:50:43.616672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76147ace-3073-439f-8c21-a31f251329e5 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:43.616960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76147ace-3073-439f-8c21-a31f251329e5 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:43.617237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76147ace-3073-439f-8c21-a31f251329e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:43.641756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-76147ace-3073-439f-8c21-a31f251329e5 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:43.642999 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-76147ace-3073-439f-8c21-a31f251329e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:43.654704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1057/2115] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:50:43.660317 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc96b53c-43df-4053-8434-17452b4860de None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:43.671493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc96b53c-43df-4053-8434-17452b4860de None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:43.676317 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6171f6bd-fd55-447f-92a4-7358adc353c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:43.676922 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6171f6bd-fd55-447f-92a4-7358adc353c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:43.677225 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6171f6bd-fd55-447f-92a4-7358adc353c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:43.677498 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6171f6bd-fd55-447f-92a4-7358adc353c9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:43.678422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6171f6bd-fd55-447f-92a4-7358adc353c9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:50:43.678726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6171f6bd-fd55-447f-92a4-7358adc353c9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:43.680510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6171f6bd-fd55-447f-92a4-7358adc353c9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:43.692566 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1059/2116] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:50:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:50:43.697069 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76147ace-3073-439f-8c21-a31f251329e5 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:43.707133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76147ace-3073-439f-8c21-a31f251329e5 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:43.711758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a9e5eb-ffe4-4dc1-8985-889fb2656b0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:43.712350 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a9e5eb-ffe4-4dc1-8985-889fb2656b0c admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:43.712626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a9e5eb-ffe4-4dc1-8985-889fb2656b0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:43.712893 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a9e5eb-ffe4-4dc1-8985-889fb2656b0c admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:43.713525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a9e5eb-ffe4-4dc1-8985-889fb2656b0c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:43.715597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a9e5eb-ffe4-4dc1-8985-889fb2656b0c admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:43.778900 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6171f6bd-fd55-447f-92a4-7358adc353c9 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:43.780778 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6171f6bd-fd55-447f-92a4-7358adc353c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:43.793421 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6171f6bd-fd55-447f-92a4-7358adc353c9 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:43.798257 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85e9edd-3436-472f-a8cb-99e265251c4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:43.798883 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85e9edd-3436-472f-a8cb-99e265251c4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:43.799223 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85e9edd-3436-472f-a8cb-99e265251c4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:43.799506 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85e9edd-3436-472f-a8cb-99e265251c4e admin admin] PATH_INFO: `/v3/users/1004952b65844462b8c500300c32d55b` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:43.800730 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85e9edd-3436-472f-a8cb-99e265251c4e admin admin] RBAC: Authorizing `identity:delete_user(user_id=1004952b65844462b8c500300c32d55b)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:43.802634 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85e9edd-3436-472f-a8cb-99e265251c4e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:43.826854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1058/2117] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:50:43 2022] POST /identity/v3/projects => generated 381 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:50:43.830801 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a9e5eb-ffe4-4dc1-8985-889fb2656b0c admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:43.892848 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a9e5eb-ffe4-4dc1-8985-889fb2656b0c admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:43.897450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4584ee2c-4c5a-42be-a247-16ccd05ccfd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:43.898043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4584ee2c-4c5a-42be-a247-16ccd05ccfd0 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:43.898323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4584ee2c-4c5a-42be-a247-16ccd05ccfd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:43.898596 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4584ee2c-4c5a-42be-a247-16ccd05ccfd0 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:43.899256 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4584ee2c-4c5a-42be-a247-16ccd05ccfd0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:43.901149 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4584ee2c-4c5a-42be-a247-16ccd05ccfd0 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:44.056508 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1059/2118] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:50:43 2022] POST /identity/v3/users => generated 291 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:50:44.062008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4584ee2c-4c5a-42be-a247-16ccd05ccfd0 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:44.072619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4584ee2c-4c5a-42be-a247-16ccd05ccfd0 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:44.077917 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a85e9edd-3436-472f-a8cb-99e265251c4e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1004952b65844462b8c500300c32d55b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:44.079918 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f00ef37-cda1-49f7-b3e9-cc6266aa778c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:44.080532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f00ef37-cda1-49f7-b3e9-cc6266aa778c admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:44.080874 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f00ef37-cda1-49f7-b3e9-cc6266aa778c admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:44.081177 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f00ef37-cda1-49f7-b3e9-cc6266aa778c admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:44.081738 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f00ef37-cda1-49f7-b3e9-cc6266aa778c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:44.083368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f00ef37-cda1-49f7-b3e9-cc6266aa778c admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:44.090182 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a85e9edd-3436-472f-a8cb-99e265251c4e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1004952b65844462b8c500300c32d55b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:44.094917 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1060/2119] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:50:44 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:50:44.098787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f00ef37-cda1-49f7-b3e9-cc6266aa778c admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:44.109092 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f00ef37-cda1-49f7-b3e9-cc6266aa778c admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:44.127545 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decd2544-02a6-4e5f-ba5d-b821f38e6f24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:44.127545 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decd2544-02a6-4e5f-ba5d-b821f38e6f24 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:44.127545 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decd2544-02a6-4e5f-ba5d-b821f38e6f24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:44.127545 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decd2544-02a6-4e5f-ba5d-b821f38e6f24 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:44.127879 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decd2544-02a6-4e5f-ba5d-b821f38e6f24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:44.127879 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decd2544-02a6-4e5f-ba5d-b821f38e6f24 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:44.129094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1061/2120] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:50:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:50:44.132954 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decd2544-02a6-4e5f-ba5d-b821f38e6f24 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:44.144165 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decd2544-02a6-4e5f-ba5d-b821f38e6f24 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:44.148824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba39574-88b9-40f3-8c3b-abe2fffa52eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:44.149468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba39574-88b9-40f3-8c3b-abe2fffa52eb admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:44.149750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba39574-88b9-40f3-8c3b-abe2fffa52eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:44.150027 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba39574-88b9-40f3-8c3b-abe2fffa52eb admin admin] PATH_INFO: `/v3/projects/0ea684aa585e4b4483261a014d861cb1/users/5691b0a2de144779b3b9a276f4f9aa79/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:44.165511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba39574-88b9-40f3-8c3b-abe2fffa52eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ea684aa585e4b4483261a014d861cb1, user_id=5691b0a2de144779b3b9a276f4f9aa79, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:44.168345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba39574-88b9-40f3-8c3b-abe2fffa52eb admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:44.192653 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a85e9edd-3436-472f-a8cb-99e265251c4e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1004952b65844462b8c500300c32d55b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:44.193304 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a85e9edd-3436-472f-a8cb-99e265251c4e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1004952b65844462b8c500300c32d55b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:44.201694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a85e9edd-3436-472f-a8cb-99e265251c4e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1004952b65844462b8c500300c32d55b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:44.207673 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1060/2121] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:50:43 2022] DELETE /identity/v3/users/1004952b65844462b8c500300c32d55b => generated 0 bytes in 430 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:50:44.212014 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e884ca3-625a-4dc6-9caa-ee2b504e9133 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:44.212300 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e884ca3-625a-4dc6-9caa-ee2b504e9133 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:44.212577 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e884ca3-625a-4dc6-9caa-ee2b504e9133 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:44.231594 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2e884ca3-625a-4dc6-9caa-ee2b504e9133 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:44.232772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2e884ca3-625a-4dc6-9caa-ee2b504e9133 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:44.279676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1062/2122] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:50:44 2022] PUT /identity/v3/projects/0ea684aa585e4b4483261a014d861cb1/users/5691b0a2de144779b3b9a276f4f9aa79/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:50:44.284325 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2437762c-421f-4649-bf72-92095d2690c6 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:44.284618 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2437762c-421f-4649-bf72-92095d2690c6 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:44.284893 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2437762c-421f-4649-bf72-92095d2690c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:44.313746 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1061/2123] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:50:44.320135 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-2437762c-421f-4649-bf72-92095d2690c6 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:44.320851 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e884ca3-625a-4dc6-9caa-ee2b504e9133 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:44.321357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-2437762c-421f-4649-bf72-92095d2690c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:44.322227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2e884ca3-625a-4dc6-9caa-ee2b504e9133 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:44.377181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:44.382034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:DELETE=4,keystone:SELECT=90,keystone:INSERT=6 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:44.401820 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1063/2124] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:50:44.415945 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2437762c-421f-4649-bf72-92095d2690c6 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:44.417215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e884ca3-625a-4dc6-9caa-ee2b504e9133 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:44.419463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-2437762c-421f-4649-bf72-92095d2690c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:44.477503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7730d71-ce32-4bb2-b1b6-3ec5d27f62cd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:44.478476 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7730d71-ce32-4bb2-b1b6-3ec5d27f62cd service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:44.479023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7730d71-ce32-4bb2-b1b6-3ec5d27f62cd service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:44.479362 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7730d71-ce32-4bb2-b1b6-3ec5d27f62cd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:44.494871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7730d71-ce32-4bb2-b1b6-3ec5d27f62cd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:44.496974 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7730d71-ce32-4bb2-b1b6-3ec5d27f62cd service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:44.502736 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2437762c-421f-4649-bf72-92095d2690c6 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:44.506469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a97533-4717-4d4c-963b-222d7df4819c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:44.506980 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1062/2125] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:50:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:50:44.507167 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a97533-4717-4d4c-963b-222d7df4819c service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:44.507449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a97533-4717-4d4c-963b-222d7df4819c service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:44.507722 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a97533-4717-4d4c-963b-222d7df4819c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:44.517775 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a97533-4717-4d4c-963b-222d7df4819c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:44.519251 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a97533-4717-4d4c-963b-222d7df4819c service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:44.529091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1064/2126] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:50:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:50:45.094997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7730d71-ce32-4bb2-b1b6-3ec5d27f62cd service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:45.096681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a7730d71-ce32-4bb2-b1b6-3ec5d27f62cd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:45.110975 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7730d71-ce32-4bb2-b1b6-3ec5d27f62cd service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:45.116140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2005340d-01be-4556-8298-8975da06f4f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:45.117349 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2005340d-01be-4556-8298-8975da06f4f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:45.117349 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2005340d-01be-4556-8298-8975da06f4f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:45.117546 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2005340d-01be-4556-8298-8975da06f4f7 admin admin] PATH_INFO: `/v3/projects/83260468ba544de7b7101469765fe9e5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:45.118860 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2005340d-01be-4556-8298-8975da06f4f7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=83260468ba544de7b7101469765fe9e5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:45.121105 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2005340d-01be-4556-8298-8975da06f4f7 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:45.269633 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2005340d-01be-4556-8298-8975da06f4f7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '83260468ba544de7b7101469765fe9e5', 'request_id': 'req-2005340d-01be-4556-8298-8975da06f4f7'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:45.276604 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2005340d-01be-4556-8298-8975da06f4f7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '83260468ba544de7b7101469765fe9e5', 'request_id': 'req-2005340d-01be-4556-8298-8975da06f4f7'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:45.398450 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2005340d-01be-4556-8298-8975da06f4f7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '83260468ba544de7b7101469765fe9e5', 'request_id': 'req-2005340d-01be-4556-8298-8975da06f4f7'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:45.412884 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2005340d-01be-4556-8298-8975da06f4f7 admin admin] The token cache is being invalidate because project 83260468ba544de7b7101469765fe9e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:50:45.413180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2005340d-01be-4556-8298-8975da06f4f7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:45.413891 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2005340d-01be-4556-8298-8975da06f4f7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:50:45.415562 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1063/2127] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:50:45 2022] DELETE /identity/v3/projects/83260468ba544de7b7101469765fe9e5 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:50:46.361047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddd49d3-9a6f-4dab-9580-205a1d016851 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:46.361546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddd49d3-9a6f-4dab-9580-205a1d016851 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:46.361796 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddd49d3-9a6f-4dab-9580-205a1d016851 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:46.393246 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-9ddd49d3-9a6f-4dab-9580-205a1d016851 None None] MFA Rules not processed for user `5691b0a2de144779b3b9a276f4f9aa79`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:46.394468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-9ddd49d3-9a6f-4dab-9580-205a1d016851 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:46.464556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1065/2128] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:46 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:50:46.476130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2005340d-01be-4556-8298-8975da06f4f7 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:46.477963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2005340d-01be-4556-8298-8975da06f4f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:46.539078 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2005340d-01be-4556-8298-8975da06f4f7 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:46.542892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f13e502-f3d7-4adf-9f46-4bb56d3efd1f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:46.543916 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f13e502-f3d7-4adf-9f46-4bb56d3efd1f service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:46.544186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f13e502-f3d7-4adf-9f46-4bb56d3efd1f service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:46.544453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f13e502-f3d7-4adf-9f46-4bb56d3efd1f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:46.555746 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f13e502-f3d7-4adf-9f46-4bb56d3efd1f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:46.557170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f13e502-f3d7-4adf-9f46-4bb56d3efd1f service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:46.567275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1064/2129] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:50:46 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:50:48.879356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc96893-992d-4fb0-8851-171311f966cb None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:48.879820 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc96893-992d-4fb0-8851-171311f966cb None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:48.879899 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc96893-992d-4fb0-8851-171311f966cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:48.898491 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-1fc96893-992d-4fb0-8851-171311f966cb None None] MFA Rules not processed for user `5691b0a2de144779b3b9a276f4f9aa79`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:48.899712 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1fc96893-992d-4fb0-8851-171311f966cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:48.915155 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1066/2130] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:48 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:50:55.568962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f13e502-f3d7-4adf-9f46-4bb56d3efd1f service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:50:55.571280 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3f13e502-f3d7-4adf-9f46-4bb56d3efd1f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:55.595895 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:50:55.650094 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f13e502-f3d7-4adf-9f46-4bb56d3efd1f service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:50:55.653939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1722d2-8719-4e21-9ef6-56e989184e00 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:50:55.655019 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1722d2-8719-4e21-9ef6-56e989184e00 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:55.655382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1722d2-8719-4e21-9ef6-56e989184e00 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:55.655821 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1722d2-8719-4e21-9ef6-56e989184e00 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:55.657835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5a1722d2-8719-4e21-9ef6-56e989184e00 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:55.730730 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1722d2-8719-4e21-9ef6-56e989184e00 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:50:55.732807 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1722d2-8719-4e21-9ef6-56e989184e00 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:50:55.743105 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1065/2131] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:50:55 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:50:58.640689 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c67acf2-f65c-4b25-a8d6-0f287e5d004e None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:50:58.640689 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c67acf2-f65c-4b25-a8d6-0f287e5d004e None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:50:58.640689 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c67acf2-f65c-4b25-a8d6-0f287e5d004e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:50:58.658480 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-1c67acf2-f65c-4b25-a8d6-0f287e5d004e None None] MFA Rules not processed for user `a6b29b9086f64ea7a7990572567642f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:50:58.659719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1c67acf2-f65c-4b25-a8d6-0f287e5d004e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:50:58.721368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1067/2132] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:50:58 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:51:02.025917 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3f6519-fdd4-485e-896e-636491d47f25 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:02.026437 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3f6519-fdd4-485e-896e-636491d47f25 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:02.026731 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3f6519-fdd4-485e-896e-636491d47f25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:02.048768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-af3f6519-fdd4-485e-896e-636491d47f25 None None] MFA Rules not processed for user `d40810f61ead4c3da414194b325da80a`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:51:02.050191 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-af3f6519-fdd4-485e-896e-636491d47f25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:02.115007 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1066/2133] 10.222.0.49 () {58 vars in 1009 bytes} [Fri Sep 16 03:51:02 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:51:02.166969 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c67acf2-f65c-4b25-a8d6-0f287e5d004e None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:02.166969 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1c67acf2-f65c-4b25-a8d6-0f287e5d004e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:02.193853 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:51:02.311654 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c67acf2-f65c-4b25-a8d6-0f287e5d004e None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:02.320345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e004269e-a56c-48fe-aee2-22f00d13c832 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa30da9972e140d7905c522576681429', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:02.321279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e004269e-a56c-48fe-aee2-22f00d13c832 service swift] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:02.321731 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e004269e-a56c-48fe-aee2-22f00d13c832 service swift] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:02.322707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e004269e-a56c-48fe-aee2-22f00d13c832 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:02.383985 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e004269e-a56c-48fe-aee2-22f00d13c832 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:02.389041 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e004269e-a56c-48fe-aee2-22f00d13c832 service swift] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:02.439865 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1068/2134] 10.222.0.49 () {60 vars in 1384 bytes} [Fri Sep 16 03:51:02 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 277 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 03:51:04.093084 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3f6519-fdd4-485e-896e-636491d47f25 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:04.094713 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-af3f6519-fdd4-485e-896e-636491d47f25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:04.180874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3f6519-fdd4-485e-896e-636491d47f25 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:04.186740 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412de37e-dcde-48e8-91a4-0d419b27b0e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:04.187517 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412de37e-dcde-48e8-91a4-0d419b27b0e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:04.187900 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412de37e-dcde-48e8-91a4-0d419b27b0e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:04.188236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412de37e-dcde-48e8-91a4-0d419b27b0e2 admin admin] PATH_INFO: `/v3/users/c65b598f95dd45938b010f4eb59959d3` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:04.189627 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412de37e-dcde-48e8-91a4-0d419b27b0e2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c65b598f95dd45938b010f4eb59959d3)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:04.192321 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412de37e-dcde-48e8-91a4-0d419b27b0e2 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:04.195912 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:51:04.293665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e21e73-753a-4626-af07-00c87124be68 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:04.294249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e21e73-753a-4626-af07-00c87124be68 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:04.295183 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e21e73-753a-4626-af07-00c87124be68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:04.322027 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-60e21e73-753a-4626-af07-00c87124be68 None None] MFA Rules not processed for user `ff7df9be8475405eba924b59a7bd6bb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:51:04.331337 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-60e21e73-753a-4626-af07-00c87124be68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:04.395173 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1069/2135] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:51:04 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:51:04.654982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-412de37e-dcde-48e8-91a4-0d419b27b0e2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c65b598f95dd45938b010f4eb59959d3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:04.683551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-412de37e-dcde-48e8-91a4-0d419b27b0e2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c65b598f95dd45938b010f4eb59959d3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:04.796381 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-412de37e-dcde-48e8-91a4-0d419b27b0e2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c65b598f95dd45938b010f4eb59959d3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:04.797119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-412de37e-dcde-48e8-91a4-0d419b27b0e2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c65b598f95dd45938b010f4eb59959d3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:04.806292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-412de37e-dcde-48e8-91a4-0d419b27b0e2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c65b598f95dd45938b010f4eb59959d3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:04.822487 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1067/2136] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:51:04 2022] DELETE /identity/v3/users/c65b598f95dd45938b010f4eb59959d3 => generated 0 bytes in 730 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:51:04.828863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f47ee9-078e-47e8-97b7-b210abc6c1ed None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:04.830751 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f47ee9-078e-47e8-97b7-b210abc6c1ed None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:04.830751 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f47ee9-078e-47e8-97b7-b210abc6c1ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:04.871534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-20f47ee9-078e-47e8-97b7-b210abc6c1ed None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:51:04.873335 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-20f47ee9-078e-47e8-97b7-b210abc6c1ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:05.037961 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1070/2137] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:51:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:51:05.062507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412de37e-dcde-48e8-91a4-0d419b27b0e2 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:05.067125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-412de37e-dcde-48e8-91a4-0d419b27b0e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:05.223041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412de37e-dcde-48e8-91a4-0d419b27b0e2 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:05.239805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3307d85-fefb-4b1e-b73a-5f7c4ba5a780 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:05.241043 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3307d85-fefb-4b1e-b73a-5f7c4ba5a780 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:05.241400 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3307d85-fefb-4b1e-b73a-5f7c4ba5a780 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:05.241754 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3307d85-fefb-4b1e-b73a-5f7c4ba5a780 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:05.258927 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3307d85-fefb-4b1e-b73a-5f7c4ba5a780 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:05.261227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3307d85-fefb-4b1e-b73a-5f7c4ba5a780 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:05.278175 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1068/2138] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:51:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:51:06.370060 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f47ee9-078e-47e8-97b7-b210abc6c1ed None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:06.372203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-20f47ee9-078e-47e8-97b7-b210abc6c1ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:06.389599 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f47ee9-078e-47e8-97b7-b210abc6c1ed None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:06.413571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2682a0-033e-4f39-afe5-15a6546c48df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:06.414234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2682a0-033e-4f39-afe5-15a6546c48df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:06.414516 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2682a0-033e-4f39-afe5-15a6546c48df admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:06.414791 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2682a0-033e-4f39-afe5-15a6546c48df admin admin] PATH_INFO: `/v3/projects/9f34b6812ea54383849ade850525bc27` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:06.424730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2682a0-033e-4f39-afe5-15a6546c48df admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f34b6812ea54383849ade850525bc27)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:06.428803 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2682a0-033e-4f39-afe5-15a6546c48df admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:06.649761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-0e2682a0-033e-4f39-afe5-15a6546c48df admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f34b6812ea54383849ade850525bc27', 'request_id': 'req-0e2682a0-033e-4f39-afe5-15a6546c48df'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:06.651981 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:51:06.724222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-0e2682a0-033e-4f39-afe5-15a6546c48df admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f34b6812ea54383849ade850525bc27', 'request_id': 'req-0e2682a0-033e-4f39-afe5-15a6546c48df'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:06.758866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-0e2682a0-033e-4f39-afe5-15a6546c48df admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f34b6812ea54383849ade850525bc27', 'request_id': 'req-0e2682a0-033e-4f39-afe5-15a6546c48df'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:06.778743 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-0e2682a0-033e-4f39-afe5-15a6546c48df admin admin] The token cache is being invalidate because project 9f34b6812ea54383849ade850525bc27 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:51:06.779659 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-0e2682a0-033e-4f39-afe5-15a6546c48df admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:06.784516 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-0e2682a0-033e-4f39-afe5-15a6546c48df admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:06.789030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1071/2139] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:51:06 2022] DELETE /identity/v3/projects/9f34b6812ea54383849ade850525bc27 => generated 0 bytes in 419 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:51:06.793516 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3307d85-fefb-4b1e-b73a-5f7c4ba5a780 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:06.804359 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3307d85-fefb-4b1e-b73a-5f7c4ba5a780 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:06.909309 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2818ae8-30e3-4d0d-be08-5fc243d35574 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:06.910429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2818ae8-30e3-4d0d-be08-5fc243d35574 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:06.911233 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2818ae8-30e3-4d0d-be08-5fc243d35574 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:06.911713 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2818ae8-30e3-4d0d-be08-5fc243d35574 admin admin] PATH_INFO: `/v3/users/f93a968411b6427abd98aaa68e324c09` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:06.913343 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2818ae8-30e3-4d0d-be08-5fc243d35574 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f93a968411b6427abd98aaa68e324c09)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:06.915307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2818ae8-30e3-4d0d-be08-5fc243d35574 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:07.351229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:51:07.569795 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c2818ae8-30e3-4d0d-be08-5fc243d35574 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f93a968411b6427abd98aaa68e324c09'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:07.601143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c2818ae8-30e3-4d0d-be08-5fc243d35574 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f93a968411b6427abd98aaa68e324c09'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:07.693927 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c2818ae8-30e3-4d0d-be08-5fc243d35574 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f93a968411b6427abd98aaa68e324c09'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:07.694672 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c2818ae8-30e3-4d0d-be08-5fc243d35574 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f93a968411b6427abd98aaa68e324c09'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:07.705256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c2818ae8-30e3-4d0d-be08-5fc243d35574 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f93a968411b6427abd98aaa68e324c09'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:07.721243 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1069/2140] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:51:06 2022] DELETE /identity/v3/users/f93a968411b6427abd98aaa68e324c09 => generated 0 bytes in 928 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:51:08.714929 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2682a0-033e-4f39-afe5-15a6546c48df admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:08.718674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-0e2682a0-033e-4f39-afe5-15a6546c48df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:08.753298 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2682a0-033e-4f39-afe5-15a6546c48df admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:08.854498 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941000f7-1b1e-4af8-bab9-c8692922fe0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:08.855268 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941000f7-1b1e-4af8-bab9-c8692922fe0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:08.855857 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941000f7-1b1e-4af8-bab9-c8692922fe0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:08.856142 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941000f7-1b1e-4af8-bab9-c8692922fe0b admin admin] PATH_INFO: `/v3/projects/56b92073e683464d83fb6182beaef0e1` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:08.857967 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941000f7-1b1e-4af8-bab9-c8692922fe0b admin admin] RBAC: Authorizing `identity:delete_project(project_id=56b92073e683464d83fb6182beaef0e1)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:08.860684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941000f7-1b1e-4af8-bab9-c8692922fe0b admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:09.002183 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-941000f7-1b1e-4af8-bab9-c8692922fe0b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '56b92073e683464d83fb6182beaef0e1', 'request_id': 'req-941000f7-1b1e-4af8-bab9-c8692922fe0b'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:09.099319 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-941000f7-1b1e-4af8-bab9-c8692922fe0b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '56b92073e683464d83fb6182beaef0e1', 'request_id': 'req-941000f7-1b1e-4af8-bab9-c8692922fe0b'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:09.108624 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-941000f7-1b1e-4af8-bab9-c8692922fe0b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '56b92073e683464d83fb6182beaef0e1', 'request_id': 'req-941000f7-1b1e-4af8-bab9-c8692922fe0b'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:09.126165 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-941000f7-1b1e-4af8-bab9-c8692922fe0b admin admin] The token cache is being invalidate because project 56b92073e683464d83fb6182beaef0e1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:51:09.126482 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-941000f7-1b1e-4af8-bab9-c8692922fe0b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:09.127178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-941000f7-1b1e-4af8-bab9-c8692922fe0b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:09.129346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1072/2141] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:51:08 2022] DELETE /identity/v3/projects/56b92073e683464d83fb6182beaef0e1 => generated 0 bytes in 415 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:51:09.137127 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de53b33-10ac-495a-af20-97c5f1c51efa None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:09.137127 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de53b33-10ac-495a-af20-97c5f1c51efa None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:09.137127 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de53b33-10ac-495a-af20-97c5f1c51efa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:09.183885 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-5de53b33-10ac-495a-af20-97c5f1c51efa None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:51:09.185877 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5de53b33-10ac-495a-af20-97c5f1c51efa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:09.320106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc133008-4d4e-416a-909b-5867a6ad0167 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:09.327067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc133008-4d4e-416a-909b-5867a6ad0167 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:09.327067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc133008-4d4e-416a-909b-5867a6ad0167 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:09.338458 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1070/2142] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:51:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:51:09.410825 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-cc133008-4d4e-416a-909b-5867a6ad0167 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:51:09.412073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-cc133008-4d4e-416a-909b-5867a6ad0167 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:09.435071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1073/2143] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:51:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:51:09.436287 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de53b33-10ac-495a-af20-97c5f1c51efa None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:09.449191 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de53b33-10ac-495a-af20-97c5f1c51efa None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:09.455730 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f84aa9-8227-473e-a3fb-64b7bbcdbd16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:09.456357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f84aa9-8227-473e-a3fb-64b7bbcdbd16 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:09.457234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f84aa9-8227-473e-a3fb-64b7bbcdbd16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:09.457511 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f84aa9-8227-473e-a3fb-64b7bbcdbd16 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:09.458288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f84aa9-8227-473e-a3fb-64b7bbcdbd16 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:51:09.458597 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f84aa9-8227-473e-a3fb-64b7bbcdbd16 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:09.460263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f84aa9-8227-473e-a3fb-64b7bbcdbd16 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:09.477733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1071/2144] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:51:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:51:09.483726 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc133008-4d4e-416a-909b-5867a6ad0167 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:09.492972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc133008-4d4e-416a-909b-5867a6ad0167 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:09.499035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41621b71-7741-472e-a51c-b7b34ee9a1fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:09.499650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41621b71-7741-472e-a51c-b7b34ee9a1fa admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:09.499934 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41621b71-7741-472e-a51c-b7b34ee9a1fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:09.500209 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41621b71-7741-472e-a51c-b7b34ee9a1fa admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:09.500846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41621b71-7741-472e-a51c-b7b34ee9a1fa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:09.502762 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41621b71-7741-472e-a51c-b7b34ee9a1fa admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:09.591678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1074/2145] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:51:09 2022] POST /identity/v3/projects => generated 379 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:51:09.597348 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f84aa9-8227-473e-a3fb-64b7bbcdbd16 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:09.608599 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f84aa9-8227-473e-a3fb-64b7bbcdbd16 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:09.674393 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6198de5-abba-4508-aff9-0d26b5506eed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:09.675083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6198de5-abba-4508-aff9-0d26b5506eed admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:09.675365 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6198de5-abba-4508-aff9-0d26b5506eed admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:09.675639 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6198de5-abba-4508-aff9-0d26b5506eed admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:09.676300 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6198de5-abba-4508-aff9-0d26b5506eed admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:09.678277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6198de5-abba-4508-aff9-0d26b5506eed admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:09.799644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1072/2146] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:51:09 2022] POST /identity/v3/users => generated 290 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:51:09.800929 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41621b71-7741-472e-a51c-b7b34ee9a1fa admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:09.818886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41621b71-7741-472e-a51c-b7b34ee9a1fa admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:09.832370 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4204d22-5f5d-4784-8e30-30d8cc0e5464 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:09.833407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4204d22-5f5d-4784-8e30-30d8cc0e5464 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:09.833696 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4204d22-5f5d-4784-8e30-30d8cc0e5464 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:09.833977 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4204d22-5f5d-4784-8e30-30d8cc0e5464 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:09.834815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4204d22-5f5d-4784-8e30-30d8cc0e5464 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:09.836506 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4204d22-5f5d-4784-8e30-30d8cc0e5464 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:09.860513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1075/2147] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:51:09 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:51:09.862215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6198de5-abba-4508-aff9-0d26b5506eed admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:09.876012 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6198de5-abba-4508-aff9-0d26b5506eed admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:09.888531 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c2b52e-a649-45ad-bec8-bfc2d10365a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:09.889034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c2b52e-a649-45ad-bec8-bfc2d10365a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:09.889314 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c2b52e-a649-45ad-bec8-bfc2d10365a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:09.889591 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c2b52e-a649-45ad-bec8-bfc2d10365a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:09.890242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c2b52e-a649-45ad-bec8-bfc2d10365a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:09.895186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c2b52e-a649-45ad-bec8-bfc2d10365a0 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:09.915381 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1073/2148] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:51:09 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:51:09.918748 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4204d22-5f5d-4784-8e30-30d8cc0e5464 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:09.947050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4204d22-5f5d-4784-8e30-30d8cc0e5464 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:09.956428 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e46cbfc-2f7e-4bf5-9a3a-c65044440d27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:09.957243 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e46cbfc-2f7e-4bf5-9a3a-c65044440d27 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:09.958142 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e46cbfc-2f7e-4bf5-9a3a-c65044440d27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:09.958259 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e46cbfc-2f7e-4bf5-9a3a-c65044440d27 admin admin] PATH_INFO: `/v3/projects/f180b132a5c649cea3ad0cc2699a04c1/users/d2bec10dc41644839b8999e19d8d46f0/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:09.980153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e46cbfc-2f7e-4bf5-9a3a-c65044440d27 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f180b132a5c649cea3ad0cc2699a04c1, user_id=d2bec10dc41644839b8999e19d8d46f0, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:09.983493 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e46cbfc-2f7e-4bf5-9a3a-c65044440d27 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:10.073071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1076/2149] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:51:09 2022] PUT /identity/v3/projects/f180b132a5c649cea3ad0cc2699a04c1/users/d2bec10dc41644839b8999e19d8d46f0/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:51:10.074791 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bee18ca-086d-4349-ae63-3234aef6509d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:10.075162 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bee18ca-086d-4349-ae63-3234aef6509d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:10.076094 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bee18ca-086d-4349-ae63-3234aef6509d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:10.110711 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:51:10.130755 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9bee18ca-086d-4349-ae63-3234aef6509d None None] MFA Rules not processed for user `d2bec10dc41644839b8999e19d8d46f0`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:51:10.132016 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9bee18ca-086d-4349-ae63-3234aef6509d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:10.231215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1074/2150] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:51:10 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:51:10.232927 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e46cbfc-2f7e-4bf5-9a3a-c65044440d27 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:10.239459 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:INSERT=4,keystone:DELETE=10,keystone:SELECT=86 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:51:10.342465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e46cbfc-2f7e-4bf5-9a3a-c65044440d27 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:10.350021 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1421fd-e112-44e3-9d4b-3e42a818c55a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:10.351039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1421fd-e112-44e3-9d4b-3e42a818c55a admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:10.353548 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1421fd-e112-44e3-9d4b-3e42a818c55a admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:10.353548 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1421fd-e112-44e3-9d4b-3e42a818c55a admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:10.353548 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1421fd-e112-44e3-9d4b-3e42a818c55a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:10.364948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1421fd-e112-44e3-9d4b-3e42a818c55a admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:10.536345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1077/2151] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:51:10 2022] POST /identity/v3/projects => generated 381 bytes in 304 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:51:10.543140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bee18ca-086d-4349-ae63-3234aef6509d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:10.572192 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bee18ca-086d-4349-ae63-3234aef6509d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:10.680230 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e697839c-5faf-4ede-bb9f-3fd78a9c4feb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:10.680859 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e697839c-5faf-4ede-bb9f-3fd78a9c4feb admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:10.682268 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e697839c-5faf-4ede-bb9f-3fd78a9c4feb admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:10.682574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e697839c-5faf-4ede-bb9f-3fd78a9c4feb admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:10.683236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e697839c-5faf-4ede-bb9f-3fd78a9c4feb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:10.687443 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e697839c-5faf-4ede-bb9f-3fd78a9c4feb admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:10.882789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1075/2152] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:51:10 2022] POST /identity/v3/users => generated 297 bytes in 340 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:51:10.886763 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1421fd-e112-44e3-9d4b-3e42a818c55a admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:10.907934 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1421fd-e112-44e3-9d4b-3e42a818c55a admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:10.920961 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa76f27-e368-475f-9fcc-6d2235f9bf16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:10.921676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa76f27-e368-475f-9fcc-6d2235f9bf16 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:10.922032 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa76f27-e368-475f-9fcc-6d2235f9bf16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:10.922384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa76f27-e368-475f-9fcc-6d2235f9bf16 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:10.923145 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa76f27-e368-475f-9fcc-6d2235f9bf16 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:10.924850 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa76f27-e368-475f-9fcc-6d2235f9bf16 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:10.947650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1078/2153] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:51:10 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:51:10.952777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e697839c-5faf-4ede-bb9f-3fd78a9c4feb admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:10.978771 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e697839c-5faf-4ede-bb9f-3fd78a9c4feb admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:10.985487 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2dfecc-8c8b-45c4-8570-f15e09178cb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:10.986238 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2dfecc-8c8b-45c4-8570-f15e09178cb1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:10.986588 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2dfecc-8c8b-45c4-8570-f15e09178cb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:10.987108 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2dfecc-8c8b-45c4-8570-f15e09178cb1 admin admin] PATH_INFO: `/v3/projects/16e81d4ee097404a995fac146f93e3cb/users/ff622fe4bcb84d78a34c196ae97c5aeb/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:11.026566 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2dfecc-8c8b-45c4-8570-f15e09178cb1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=16e81d4ee097404a995fac146f93e3cb, user_id=ff622fe4bcb84d78a34c196ae97c5aeb, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:11.035734 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2dfecc-8c8b-45c4-8570-f15e09178cb1 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:11.184370 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1076/2154] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:51:10 2022] PUT /identity/v3/projects/16e81d4ee097404a995fac146f93e3cb/users/ff622fe4bcb84d78a34c196ae97c5aeb/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:51:11.188653 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa76f27-e368-475f-9fcc-6d2235f9bf16 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:11.211833 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa76f27-e368-475f-9fcc-6d2235f9bf16 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:11.360428 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14e4031-67b4-4c28-bf8e-04715e7cb9be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:11.363037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14e4031-67b4-4c28-bf8e-04715e7cb9be admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:11.363037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14e4031-67b4-4c28-bf8e-04715e7cb9be admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:11.363037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14e4031-67b4-4c28-bf8e-04715e7cb9be admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:11.364831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14e4031-67b4-4c28-bf8e-04715e7cb9be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:11.366696 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14e4031-67b4-4c28-bf8e-04715e7cb9be admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:11.383284 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2dfecc-8c8b-45c4-8570-f15e09178cb1 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:11.391122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1079/2155] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:51:11 2022] GET /identity/v3/roles => generated 1420 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:51:11.402288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2dfecc-8c8b-45c4-8570-f15e09178cb1 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:11.417231 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a5c02a-1b17-4128-9eb4-51f5bbba67c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:11.417923 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a5c02a-1b17-4128-9eb4-51f5bbba67c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:11.420198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a5c02a-1b17-4128-9eb4-51f5bbba67c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:11.420198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a5c02a-1b17-4128-9eb4-51f5bbba67c2 admin admin] PATH_INFO: `/v3/projects/16e81d4ee097404a995fac146f93e3cb/users/ff622fe4bcb84d78a34c196ae97c5aeb/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:11.422623 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a5c02a-1b17-4128-9eb4-51f5bbba67c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=16e81d4ee097404a995fac146f93e3cb, user_id=ff622fe4bcb84d78a34c196ae97c5aeb, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:11.435235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a5c02a-1b17-4128-9eb4-51f5bbba67c2 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:11.461698 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1077/2156] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:51:11 2022] PUT /identity/v3/projects/16e81d4ee097404a995fac146f93e3cb/users/ff622fe4bcb84d78a34c196ae97c5aeb/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:51:11.472654 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e224f548-a124-47a8-8292-de04a2b7fc37 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:11.473024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e224f548-a124-47a8-8292-de04a2b7fc37 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:11.473414 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e224f548-a124-47a8-8292-de04a2b7fc37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:11.509379 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-e224f548-a124-47a8-8292-de04a2b7fc37 None None] MFA Rules not processed for user `ff622fe4bcb84d78a34c196ae97c5aeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:51:11.510422 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e224f548-a124-47a8-8292-de04a2b7fc37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:11.640727 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:51:11.648778 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1080/2157] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:51:11 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:51:11.664423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a5c02a-1b17-4128-9eb4-51f5bbba67c2 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:11.667285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-10a5c02a-1b17-4128-9eb4-51f5bbba67c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:11.788909 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a5c02a-1b17-4128-9eb4-51f5bbba67c2 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:11.803218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e57782-cfc4-43ba-91d3-a0969258ca74 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:11.804324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e57782-cfc4-43ba-91d3-a0969258ca74 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:11.804718 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e57782-cfc4-43ba-91d3-a0969258ca74 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:11.805092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e57782-cfc4-43ba-91d3-a0969258ca74 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:11.825682 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e57782-cfc4-43ba-91d3-a0969258ca74 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:11.828895 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:51:11.834626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e57782-cfc4-43ba-91d3-a0969258ca74 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:11.854488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1078/2158] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:51:11 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:51:13.632333 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e224f548-a124-47a8-8292-de04a2b7fc37 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:13.649058 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e224f548-a124-47a8-8292-de04a2b7fc37 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:13.659948 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320f056b-11c2-4bf3-b32b-de0fae2e7f1f tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ff622fe4bcb84d78a34c196ae97c5aeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '16e81d4ee097404a995fac146f93e3cb', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:13.660594 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320f056b-11c2-4bf3-b32b-de0fae2e7f1f tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:13.660887 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320f056b-11c2-4bf3-b32b-de0fae2e7f1f tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:13.661168 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320f056b-11c2-4bf3-b32b-de0fae2e7f1f tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:13.661945 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320f056b-11c2-4bf3-b32b-de0fae2e7f1f tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:51:13.662260 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320f056b-11c2-4bf3-b32b-de0fae2e7f1f tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:13.663903 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320f056b-11c2-4bf3-b32b-de0fae2e7f1f tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:13.694581 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1081/2159] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:51:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:51:13.696256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e57782-cfc4-43ba-91d3-a0969258ca74 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:13.713864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e57782-cfc4-43ba-91d3-a0969258ca74 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:13.731645 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93bfef7-95bd-4f21-ba60-85818e827c76 tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ff622fe4bcb84d78a34c196ae97c5aeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '16e81d4ee097404a995fac146f93e3cb', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:13.732474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93bfef7-95bd-4f21-ba60-85818e827c76 tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:13.732754 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93bfef7-95bd-4f21-ba60-85818e827c76 tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:13.733036 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93bfef7-95bd-4f21-ba60-85818e827c76 tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:13.733752 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93bfef7-95bd-4f21-ba60-85818e827c76 tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:13.737472 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93bfef7-95bd-4f21-ba60-85818e827c76 tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:13.902177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1079/2160] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:51:13 2022] POST /identity/v3/projects => generated 353 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:51:13.909028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320f056b-11c2-4bf3-b32b-de0fae2e7f1f tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:13.935407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320f056b-11c2-4bf3-b32b-de0fae2e7f1f tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:14.034463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b8fa95-64f9-4ed6-82c5-3bfc8950284a tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ff622fe4bcb84d78a34c196ae97c5aeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '16e81d4ee097404a995fac146f93e3cb', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:14.040094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b8fa95-64f9-4ed6-82c5-3bfc8950284a tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:14.040094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b8fa95-64f9-4ed6-82c5-3bfc8950284a tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:14.040094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b8fa95-64f9-4ed6-82c5-3bfc8950284a tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:14.040094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b8fa95-64f9-4ed6-82c5-3bfc8950284a tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:51:14.040094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b8fa95-64f9-4ed6-82c5-3bfc8950284a tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:14.040780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b8fa95-64f9-4ed6-82c5-3bfc8950284a tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:14.057336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1082/2161] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:51:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:51:14.108657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93bfef7-95bd-4f21-ba60-85818e827c76 tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:14.124930 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93bfef7-95bd-4f21-ba60-85818e827c76 tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:14.135728 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4367fdd-3c99-477c-82d9-06fe66d389df tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ff622fe4bcb84d78a34c196ae97c5aeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '16e81d4ee097404a995fac146f93e3cb', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:14.136644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4367fdd-3c99-477c-82d9-06fe66d389df tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:14.137110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4367fdd-3c99-477c-82d9-06fe66d389df tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:14.137713 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4367fdd-3c99-477c-82d9-06fe66d389df tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] PATH_INFO: `/v3/projects/d7ea5c4a0232468180ece38b5d09b95b` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:14.139176 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4367fdd-3c99-477c-82d9-06fe66d389df tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] RBAC: Authorizing `identity:delete_project(project_id=d7ea5c4a0232468180ece38b5d09b95b)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:14.141942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4367fdd-3c99-477c-82d9-06fe66d389df tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:14.377304 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a4367fdd-3c99-477c-82d9-06fe66d389df tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd7ea5c4a0232468180ece38b5d09b95b', 'request_id': 'req-a4367fdd-3c99-477c-82d9-06fe66d389df'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:14.392898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a4367fdd-3c99-477c-82d9-06fe66d389df tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd7ea5c4a0232468180ece38b5d09b95b', 'request_id': 'req-a4367fdd-3c99-477c-82d9-06fe66d389df'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:14.495641 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a4367fdd-3c99-477c-82d9-06fe66d389df tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd7ea5c4a0232468180ece38b5d09b95b', 'request_id': 'req-a4367fdd-3c99-477c-82d9-06fe66d389df'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:14.519655 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a4367fdd-3c99-477c-82d9-06fe66d389df tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] The token cache is being invalidate because project d7ea5c4a0232468180ece38b5d09b95b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:51:14.521756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a4367fdd-3c99-477c-82d9-06fe66d389df tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:14.524576 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a4367fdd-3c99-477c-82d9-06fe66d389df tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:14.528676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1080/2162] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:51:14 2022] DELETE /identity/v3/projects/d7ea5c4a0232468180ece38b5d09b95b => generated 0 bytes in 420 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:51:15.442259 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b8fa95-64f9-4ed6-82c5-3bfc8950284a tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:15.444770 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-33b8fa95-64f9-4ed6-82c5-3bfc8950284a tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:15.576864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b8fa95-64f9-4ed6-82c5-3bfc8950284a tempest-VolumeQuotaClassesTest-1046650038 tempest-VolumeQuotaClassesTest-1046650038-project-admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:15.739485 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ba26f8-4b4e-4975-bf1c-5ecd5a7c4fce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:15.739714 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ba26f8-4b4e-4975-bf1c-5ecd5a7c4fce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:15.739937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ba26f8-4b4e-4975-bf1c-5ecd5a7c4fce admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:15.740224 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ba26f8-4b4e-4975-bf1c-5ecd5a7c4fce admin admin] PATH_INFO: `/v3/users/d2bec10dc41644839b8999e19d8d46f0` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:15.742273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ba26f8-4b4e-4975-bf1c-5ecd5a7c4fce admin admin] RBAC: Authorizing `identity:delete_user(user_id=d2bec10dc41644839b8999e19d8d46f0)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:15.745093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ba26f8-4b4e-4975-bf1c-5ecd5a7c4fce admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:16.012073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:51:16.037750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-28ba26f8-4b4e-4975-bf1c-5ecd5a7c4fce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd2bec10dc41644839b8999e19d8d46f0'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:16.130531 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-28ba26f8-4b4e-4975-bf1c-5ecd5a7c4fce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd2bec10dc41644839b8999e19d8d46f0'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:16.147285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-28ba26f8-4b4e-4975-bf1c-5ecd5a7c4fce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd2bec10dc41644839b8999e19d8d46f0'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:16.148102 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-28ba26f8-4b4e-4975-bf1c-5ecd5a7c4fce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd2bec10dc41644839b8999e19d8d46f0'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:16.168014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-28ba26f8-4b4e-4975-bf1c-5ecd5a7c4fce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd2bec10dc41644839b8999e19d8d46f0'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:16.183457 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1083/2163] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:51:15 2022] DELETE /identity/v3/users/d2bec10dc41644839b8999e19d8d46f0 => generated 0 bytes in 742 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:51:16.189078 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e72e9b6-1635-4373-a7b6-632e0bf88e8d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:16.190053 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e72e9b6-1635-4373-a7b6-632e0bf88e8d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:16.190565 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e72e9b6-1635-4373-a7b6-632e0bf88e8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:16.222878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1e72e9b6-1635-4373-a7b6-632e0bf88e8d None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:51:16.228337 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1e72e9b6-1635-4373-a7b6-632e0bf88e8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:16.344825 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1081/2164] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:51:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:51:16.352705 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ba26f8-4b4e-4975-bf1c-5ecd5a7c4fce admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:16.354346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-28ba26f8-4b4e-4975-bf1c-5ecd5a7c4fce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:16.456443 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ba26f8-4b4e-4975-bf1c-5ecd5a7c4fce admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:16.464901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b6d543-81b1-4235-bdd5-5bc45b6d7fcb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:16.465914 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b6d543-81b1-4235-bdd5-5bc45b6d7fcb service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:16.466933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b6d543-81b1-4235-bdd5-5bc45b6d7fcb service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:16.467367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b6d543-81b1-4235-bdd5-5bc45b6d7fcb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:16.493243 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b6d543-81b1-4235-bdd5-5bc45b6d7fcb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:16.497081 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b6d543-81b1-4235-bdd5-5bc45b6d7fcb service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:16.511230 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1084/2165] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:51:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:51:17.850001 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e72e9b6-1635-4373-a7b6-632e0bf88e8d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:17.851590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1e72e9b6-1635-4373-a7b6-632e0bf88e8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:17.868659 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e72e9b6-1635-4373-a7b6-632e0bf88e8d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:17.875451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d4e4f5-c81e-4e19-873f-418e0b79e3df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:17.875451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d4e4f5-c81e-4e19-873f-418e0b79e3df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:17.875869 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d4e4f5-c81e-4e19-873f-418e0b79e3df admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:17.876039 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d4e4f5-c81e-4e19-873f-418e0b79e3df admin admin] PATH_INFO: `/v3/projects/f180b132a5c649cea3ad0cc2699a04c1` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:17.877208 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d4e4f5-c81e-4e19-873f-418e0b79e3df admin admin] RBAC: Authorizing `identity:delete_project(project_id=f180b132a5c649cea3ad0cc2699a04c1)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:17.888019 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d4e4f5-c81e-4e19-873f-418e0b79e3df admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:18.066126 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a8d4e4f5-c81e-4e19-873f-418e0b79e3df admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f180b132a5c649cea3ad0cc2699a04c1', 'request_id': 'req-a8d4e4f5-c81e-4e19-873f-418e0b79e3df'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:18.067589 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:51:18.078240 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a8d4e4f5-c81e-4e19-873f-418e0b79e3df admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f180b132a5c649cea3ad0cc2699a04c1', 'request_id': 'req-a8d4e4f5-c81e-4e19-873f-418e0b79e3df'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:18.184788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a8d4e4f5-c81e-4e19-873f-418e0b79e3df admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f180b132a5c649cea3ad0cc2699a04c1', 'request_id': 'req-a8d4e4f5-c81e-4e19-873f-418e0b79e3df'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:18.201499 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a8d4e4f5-c81e-4e19-873f-418e0b79e3df admin admin] The token cache is being invalidate because project f180b132a5c649cea3ad0cc2699a04c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:51:18.201499 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a8d4e4f5-c81e-4e19-873f-418e0b79e3df admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:18.204915 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a8d4e4f5-c81e-4e19-873f-418e0b79e3df admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:18.210143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1082/2166] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:51:17 2022] DELETE /identity/v3/projects/f180b132a5c649cea3ad0cc2699a04c1 => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:51:18.211638 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b6d543-81b1-4235-bdd5-5bc45b6d7fcb service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:18.226952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b6d543-81b1-4235-bdd5-5bc45b6d7fcb service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:18.326313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d9e0af-3096-439c-a0f5-451addde3234 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:18.327338 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d9e0af-3096-439c-a0f5-451addde3234 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:18.327868 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d9e0af-3096-439c-a0f5-451addde3234 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:18.330528 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d9e0af-3096-439c-a0f5-451addde3234 admin admin] PATH_INFO: `/v3/users/ff622fe4bcb84d78a34c196ae97c5aeb` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:18.332000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d9e0af-3096-439c-a0f5-451addde3234 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ff622fe4bcb84d78a34c196ae97c5aeb)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:18.334456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d9e0af-3096-439c-a0f5-451addde3234 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:18.581871 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-19d9e0af-3096-439c-a0f5-451addde3234 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ff622fe4bcb84d78a34c196ae97c5aeb'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:18.591547 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:51:18.707953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-19d9e0af-3096-439c-a0f5-451addde3234 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ff622fe4bcb84d78a34c196ae97c5aeb'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:18.732002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-19d9e0af-3096-439c-a0f5-451addde3234 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ff622fe4bcb84d78a34c196ae97c5aeb'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:18.733611 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-19d9e0af-3096-439c-a0f5-451addde3234 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ff622fe4bcb84d78a34c196ae97c5aeb'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:18.754419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-19d9e0af-3096-439c-a0f5-451addde3234 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ff622fe4bcb84d78a34c196ae97c5aeb'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:18.764675 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1085/2167] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:51:18 2022] DELETE /identity/v3/users/ff622fe4bcb84d78a34c196ae97c5aeb => generated 0 bytes in 554 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:51:19.730224 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d4e4f5-c81e-4e19-873f-418e0b79e3df admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:19.735867 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a8d4e4f5-c81e-4e19-873f-418e0b79e3df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:19.751386 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d4e4f5-c81e-4e19-873f-418e0b79e3df admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:19.869660 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508229e0-1418-4c5a-974d-94f3672c02f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:19.870888 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508229e0-1418-4c5a-974d-94f3672c02f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:19.871366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508229e0-1418-4c5a-974d-94f3672c02f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:19.871646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508229e0-1418-4c5a-974d-94f3672c02f9 admin admin] PATH_INFO: `/v3/projects/16e81d4ee097404a995fac146f93e3cb` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:19.873197 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508229e0-1418-4c5a-974d-94f3672c02f9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=16e81d4ee097404a995fac146f93e3cb)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:19.875877 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508229e0-1418-4c5a-974d-94f3672c02f9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:20.015588 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-508229e0-1418-4c5a-974d-94f3672c02f9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '16e81d4ee097404a995fac146f93e3cb', 'request_id': 'req-508229e0-1418-4c5a-974d-94f3672c02f9'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:20.028786 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-508229e0-1418-4c5a-974d-94f3672c02f9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '16e81d4ee097404a995fac146f93e3cb', 'request_id': 'req-508229e0-1418-4c5a-974d-94f3672c02f9'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:20.129127 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-508229e0-1418-4c5a-974d-94f3672c02f9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '16e81d4ee097404a995fac146f93e3cb', 'request_id': 'req-508229e0-1418-4c5a-974d-94f3672c02f9'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:20.140879 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-508229e0-1418-4c5a-974d-94f3672c02f9 admin admin] The token cache is being invalidate because project 16e81d4ee097404a995fac146f93e3cb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:51:20.141438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-508229e0-1418-4c5a-974d-94f3672c02f9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:20.142261 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-508229e0-1418-4c5a-974d-94f3672c02f9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:20.144201 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1083/2168] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:51:19 2022] DELETE /identity/v3/projects/16e81d4ee097404a995fac146f93e3cb => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:51:20.149399 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9bd212-9f9e-4e30-9985-14cfabba04b6 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:20.150020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9bd212-9f9e-4e30-9985-14cfabba04b6 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:20.150020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9bd212-9f9e-4e30-9985-14cfabba04b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:20.175106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-7e9bd212-9f9e-4e30-9985-14cfabba04b6 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:51:20.176458 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-7e9bd212-9f9e-4e30-9985-14cfabba04b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:20.268657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1086/2169] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:51:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:51:20.277060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed3a0f4-d541-49e1-9427-7a7e2a609fa2 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:20.277352 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed3a0f4-d541-49e1-9427-7a7e2a609fa2 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:20.277630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed3a0f4-d541-49e1-9427-7a7e2a609fa2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:20.301309 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-fed3a0f4-d541-49e1-9427-7a7e2a609fa2 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:51:20.302545 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fed3a0f4-d541-49e1-9427-7a7e2a609fa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:20.316260 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1084/2170] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:51:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:51:20.320530 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9bd212-9f9e-4e30-9985-14cfabba04b6 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:20.332558 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9bd212-9f9e-4e30-9985-14cfabba04b6 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:20.338052 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521a8154-f4b0-4e42-81ec-7a40237ed120 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:20.338939 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521a8154-f4b0-4e42-81ec-7a40237ed120 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:20.339586 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521a8154-f4b0-4e42-81ec-7a40237ed120 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:20.339586 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521a8154-f4b0-4e42-81ec-7a40237ed120 admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:20.340500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521a8154-f4b0-4e42-81ec-7a40237ed120 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:51:20.341093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521a8154-f4b0-4e42-81ec-7a40237ed120 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:20.342940 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521a8154-f4b0-4e42-81ec-7a40237ed120 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:20.361235 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1087/2171] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:51:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:51:20.362076 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed3a0f4-d541-49e1-9427-7a7e2a609fa2 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:20.373619 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed3a0f4-d541-49e1-9427-7a7e2a609fa2 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:20.379974 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ac22fc-4609-4741-a951-06dd8972b557 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:20.380227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ac22fc-4609-4741-a951-06dd8972b557 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:20.380747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ac22fc-4609-4741-a951-06dd8972b557 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:20.381123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ac22fc-4609-4741-a951-06dd8972b557 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:20.381931 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ac22fc-4609-4741-a951-06dd8972b557 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:20.384029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ac22fc-4609-4741-a951-06dd8972b557 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:20.522742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1085/2172] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:51:20 2022] POST /identity/v3/projects => generated 385 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:51:20.526999 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521a8154-f4b0-4e42-81ec-7a40237ed120 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:20.541012 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521a8154-f4b0-4e42-81ec-7a40237ed120 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:20.605185 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac09d00d-f4b1-4bfb-98b0-1d46eac2d1ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:20.606045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac09d00d-f4b1-4bfb-98b0-1d46eac2d1ca admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:20.606354 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac09d00d-f4b1-4bfb-98b0-1d46eac2d1ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:20.607312 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac09d00d-f4b1-4bfb-98b0-1d46eac2d1ca admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:20.608329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac09d00d-f4b1-4bfb-98b0-1d46eac2d1ca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:20.610821 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac09d00d-f4b1-4bfb-98b0-1d46eac2d1ca admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:20.718795 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1088/2173] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:51:20 2022] POST /identity/v3/users => generated 293 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:51:20.723013 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ac22fc-4609-4741-a951-06dd8972b557 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:20.734715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ac22fc-4609-4741-a951-06dd8972b557 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:20.742037 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57615736-3c6d-43d7-9969-0679be6848fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:20.742818 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57615736-3c6d-43d7-9969-0679be6848fa admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:20.743141 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57615736-3c6d-43d7-9969-0679be6848fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:20.743403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57615736-3c6d-43d7-9969-0679be6848fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:20.744058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57615736-3c6d-43d7-9969-0679be6848fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:20.745665 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57615736-3c6d-43d7-9969-0679be6848fa admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:20.756651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1086/2174] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:51:20 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:51:20.760726 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac09d00d-f4b1-4bfb-98b0-1d46eac2d1ca admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:20.770726 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac09d00d-f4b1-4bfb-98b0-1d46eac2d1ca admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:20.775393 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a3c334-8aa3-42ba-b131-a775c643055d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:20.775992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a3c334-8aa3-42ba-b131-a775c643055d admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:20.776272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a3c334-8aa3-42ba-b131-a775c643055d admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:20.776545 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a3c334-8aa3-42ba-b131-a775c643055d admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:20.777184 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a3c334-8aa3-42ba-b131-a775c643055d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:20.779486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a3c334-8aa3-42ba-b131-a775c643055d admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:20.790515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1089/2175] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:51:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:51:20.795112 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57615736-3c6d-43d7-9969-0679be6848fa admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:20.805341 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57615736-3c6d-43d7-9969-0679be6848fa admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:20.810171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b588a770-2297-4405-8453-d967bb29e27a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:20.810838 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b588a770-2297-4405-8453-d967bb29e27a admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:20.811165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b588a770-2297-4405-8453-d967bb29e27a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:20.811479 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b588a770-2297-4405-8453-d967bb29e27a admin admin] PATH_INFO: `/v3/projects/1000f2f2f1a74fdf9a88f71b3ba6302e/users/183649a4d2484e04bbd70e6d1f6f9d49/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:20.826680 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b588a770-2297-4405-8453-d967bb29e27a admin admin] RBAC: Authorizing `identity:create_grant(project_id=1000f2f2f1a74fdf9a88f71b3ba6302e, user_id=183649a4d2484e04bbd70e6d1f6f9d49, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:20.829341 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b588a770-2297-4405-8453-d967bb29e27a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:20.908994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1087/2176] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:51:20 2022] PUT /identity/v3/projects/1000f2f2f1a74fdf9a88f71b3ba6302e/users/183649a4d2484e04bbd70e6d1f6f9d49/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:51:20.919274 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e278ce-836d-4420-8784-acdba163dbb1 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:20.920363 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e278ce-836d-4420-8784-acdba163dbb1 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:20.920676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e278ce-836d-4420-8784-acdba163dbb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:20.963910 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-81e278ce-836d-4420-8784-acdba163dbb1 None None] MFA Rules not processed for user `183649a4d2484e04bbd70e6d1f6f9d49`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:51:20.965447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-81e278ce-836d-4420-8784-acdba163dbb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:20.993007 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:51:21.014757 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b588a770-2297-4405-8453-d967bb29e27a admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:21.016626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b588a770-2297-4405-8453-d967bb29e27a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:21.023868 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:51:21.041639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1090/2177] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:51:20 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:51:21.053882 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e278ce-836d-4420-8784-acdba163dbb1 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:21.055398 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-81e278ce-836d-4420-8784-acdba163dbb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:21.101809 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b588a770-2297-4405-8453-d967bb29e27a admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:21.109635 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd82a95-f44b-4ae1-9569-311c3731af61 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '176da4679fcb46148da9289fccdad5c2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:21.111314 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd82a95-f44b-4ae1-9569-311c3731af61 service placement] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:21.111314 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd82a95-f44b-4ae1-9569-311c3731af61 service placement] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:21.111314 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd82a95-f44b-4ae1-9569-311c3731af61 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:21.112637 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8dd82a95-f44b-4ae1-9569-311c3731af61 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:21.123698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e278ce-836d-4420-8784-acdba163dbb1 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:21.124602 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd82a95-f44b-4ae1-9569-311c3731af61 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:21.126186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd82a95-f44b-4ae1-9569-311c3731af61 service placement] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:21.128104 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9274f978-66e0-404c-be8f-d254f8176f8b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:21.128515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9274f978-66e0-404c-be8f-d254f8176f8b service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:21.131843 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9274f978-66e0-404c-be8f-d254f8176f8b service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:21.131843 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9274f978-66e0-404c-be8f-d254f8176f8b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:21.144051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1088/2178] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:51:21 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:51:21.144786 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9274f978-66e0-404c-be8f-d254f8176f8b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:21.147502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9274f978-66e0-404c-be8f-d254f8176f8b service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:21.158398 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1091/2179] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:51:21 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:51:21.178435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd82a95-f44b-4ae1-9569-311c3731af61 service placement] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:21.257699 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd82a95-f44b-4ae1-9569-311c3731af61 service placement] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:21.264392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496ae20c-6859-481a-812b-5d05f215f2d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:21.265242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496ae20c-6859-481a-812b-5d05f215f2d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:21.265595 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496ae20c-6859-481a-812b-5d05f215f2d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:21.266378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496ae20c-6859-481a-812b-5d05f215f2d5 admin admin] PATH_INFO: `/v3/users/183649a4d2484e04bbd70e6d1f6f9d49` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:21.267803 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496ae20c-6859-481a-812b-5d05f215f2d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=183649a4d2484e04bbd70e6d1f6f9d49)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:21.271711 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496ae20c-6859-481a-812b-5d05f215f2d5 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:21.503353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-496ae20c-6859-481a-812b-5d05f215f2d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '183649a4d2484e04bbd70e6d1f6f9d49'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:21.518665 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-496ae20c-6859-481a-812b-5d05f215f2d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '183649a4d2484e04bbd70e6d1f6f9d49'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:21.616020 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-496ae20c-6859-481a-812b-5d05f215f2d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '183649a4d2484e04bbd70e6d1f6f9d49'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:21.616640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-496ae20c-6859-481a-812b-5d05f215f2d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '183649a4d2484e04bbd70e6d1f6f9d49'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:21.624512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-496ae20c-6859-481a-812b-5d05f215f2d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '183649a4d2484e04bbd70e6d1f6f9d49'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:21.630416 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1089/2180] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:51:21 2022] DELETE /identity/v3/users/183649a4d2484e04bbd70e6d1f6f9d49 => generated 0 bytes in 453 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:51:21.634800 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a68d786-b441-4267-8746-9de46de32909 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:21.635179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a68d786-b441-4267-8746-9de46de32909 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:21.635456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a68d786-b441-4267-8746-9de46de32909 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:21.654864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-5a68d786-b441-4267-8746-9de46de32909 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:51:21.659791 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-5a68d786-b441-4267-8746-9de46de32909 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:21.725460 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1092/2181] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:51:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:51:21.736010 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496ae20c-6859-481a-812b-5d05f215f2d5 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:21.737585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-496ae20c-6859-481a-812b-5d05f215f2d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:21.757211 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:51:21.801890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496ae20c-6859-481a-812b-5d05f215f2d5 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:21.805826 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd08dd8-0d14-4eeb-a54c-0b4ae2924b19 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:21.806465 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd08dd8-0d14-4eeb-a54c-0b4ae2924b19 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:21.806773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd08dd8-0d14-4eeb-a54c-0b4ae2924b19 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:21.807077 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd08dd8-0d14-4eeb-a54c-0b4ae2924b19 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:21.817444 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd08dd8-0d14-4eeb-a54c-0b4ae2924b19 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:21.819302 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd08dd8-0d14-4eeb-a54c-0b4ae2924b19 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:21.830875 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1090/2182] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:51:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:51:22.271106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a68d786-b441-4267-8746-9de46de32909 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:22.271106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-5a68d786-b441-4267-8746-9de46de32909 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:22.273708 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a68d786-b441-4267-8746-9de46de32909 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:22.282226 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda6a8e7-f0c9-4d32-8293-0f554f77839c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:22.282226 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda6a8e7-f0c9-4d32-8293-0f554f77839c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:22.282226 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda6a8e7-f0c9-4d32-8293-0f554f77839c admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:22.282802 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda6a8e7-f0c9-4d32-8293-0f554f77839c admin admin] PATH_INFO: `/v3/projects/1000f2f2f1a74fdf9a88f71b3ba6302e` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:22.283799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda6a8e7-f0c9-4d32-8293-0f554f77839c admin admin] RBAC: Authorizing `identity:delete_project(project_id=1000f2f2f1a74fdf9a88f71b3ba6302e)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:22.285710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda6a8e7-f0c9-4d32-8293-0f554f77839c admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:22.389238 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-eda6a8e7-f0c9-4d32-8293-0f554f77839c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1000f2f2f1a74fdf9a88f71b3ba6302e', 'request_id': 'req-eda6a8e7-f0c9-4d32-8293-0f554f77839c'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:22.466997 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-eda6a8e7-f0c9-4d32-8293-0f554f77839c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1000f2f2f1a74fdf9a88f71b3ba6302e', 'request_id': 'req-eda6a8e7-f0c9-4d32-8293-0f554f77839c'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:22.472643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:51:22.480202 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-eda6a8e7-f0c9-4d32-8293-0f554f77839c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1000f2f2f1a74fdf9a88f71b3ba6302e', 'request_id': 'req-eda6a8e7-f0c9-4d32-8293-0f554f77839c'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:22.494142 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-eda6a8e7-f0c9-4d32-8293-0f554f77839c admin admin] The token cache is being invalidate because project 1000f2f2f1a74fdf9a88f71b3ba6302e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:51:22.494435 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-eda6a8e7-f0c9-4d32-8293-0f554f77839c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:22.495246 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-eda6a8e7-f0c9-4d32-8293-0f554f77839c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:22.496827 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1093/2183] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:51:22 2022] DELETE /identity/v3/projects/1000f2f2f1a74fdf9a88f71b3ba6302e => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:51:22.502384 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941d517b-285b-4968-b7c7-8a55a4eb7ec6 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:22.502762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941d517b-285b-4968-b7c7-8a55a4eb7ec6 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:22.503147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941d517b-285b-4968-b7c7-8a55a4eb7ec6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:22.523858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-941d517b-285b-4968-b7c7-8a55a4eb7ec6 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:51:22.524916 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-941d517b-285b-4968-b7c7-8a55a4eb7ec6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:22.592373 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1091/2184] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:51:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:51:22.600778 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c480738-5182-4fba-99aa-0b4222e8441b None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:22.601132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c480738-5182-4fba-99aa-0b4222e8441b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:22.601437 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c480738-5182-4fba-99aa-0b4222e8441b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:22.621170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-2c480738-5182-4fba-99aa-0b4222e8441b None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:51:22.623043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-2c480738-5182-4fba-99aa-0b4222e8441b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:22.630983 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1094/2185] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:51:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:51:22.635149 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941d517b-285b-4968-b7c7-8a55a4eb7ec6 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:22.644778 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941d517b-285b-4968-b7c7-8a55a4eb7ec6 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:22.650619 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d5f167-9006-4de7-97a6-f7990b6c2f09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:22.651313 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d5f167-9006-4de7-97a6-f7990b6c2f09 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:22.651590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d5f167-9006-4de7-97a6-f7990b6c2f09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:22.651878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d5f167-9006-4de7-97a6-f7990b6c2f09 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:22.652503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d5f167-9006-4de7-97a6-f7990b6c2f09 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:51:22.652802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d5f167-9006-4de7-97a6-f7990b6c2f09 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:22.654560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d5f167-9006-4de7-97a6-f7990b6c2f09 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:22.667110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1092/2186] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:51:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:51:22.671765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c480738-5182-4fba-99aa-0b4222e8441b None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:22.681163 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c480738-5182-4fba-99aa-0b4222e8441b None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:22.686399 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5180ba99-c1dd-4129-96ac-f22e37d7cbdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:22.687032 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5180ba99-c1dd-4129-96ac-f22e37d7cbdd admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:22.687329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5180ba99-c1dd-4129-96ac-f22e37d7cbdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:22.687599 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5180ba99-c1dd-4129-96ac-f22e37d7cbdd admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:22.688225 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5180ba99-c1dd-4129-96ac-f22e37d7cbdd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:22.690114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5180ba99-c1dd-4129-96ac-f22e37d7cbdd admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:22.774327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1095/2187] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:51:22 2022] POST /identity/v3/projects => generated 371 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:51:22.778409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d5f167-9006-4de7-97a6-f7990b6c2f09 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:22.788311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d5f167-9006-4de7-97a6-f7990b6c2f09 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:22.843952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc88eb7-1ebc-4862-8303-d8a5f168dae7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:22.844556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc88eb7-1ebc-4862-8303-d8a5f168dae7 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:22.844853 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc88eb7-1ebc-4862-8303-d8a5f168dae7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:22.845105 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc88eb7-1ebc-4862-8303-d8a5f168dae7 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:22.845734 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc88eb7-1ebc-4862-8303-d8a5f168dae7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:22.847708 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc88eb7-1ebc-4862-8303-d8a5f168dae7 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:22.930840 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1093/2188] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:51:22 2022] POST /identity/v3/users => generated 286 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:51:22.932104 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5180ba99-c1dd-4129-96ac-f22e37d7cbdd admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:22.942833 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5180ba99-c1dd-4129-96ac-f22e37d7cbdd admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:22.947582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c428bdb-a836-4b69-a689-e8ca34bd13cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:22.948185 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c428bdb-a836-4b69-a689-e8ca34bd13cc admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:22.948473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c428bdb-a836-4b69-a689-e8ca34bd13cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:22.948741 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c428bdb-a836-4b69-a689-e8ca34bd13cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:22.949411 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c428bdb-a836-4b69-a689-e8ca34bd13cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:22.952029 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c428bdb-a836-4b69-a689-e8ca34bd13cc admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:22.965366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1096/2189] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:51:22 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:51:22.968837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc88eb7-1ebc-4862-8303-d8a5f168dae7 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:22.979114 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc88eb7-1ebc-4862-8303-d8a5f168dae7 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:22.984300 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c5c333-9a95-4772-9faa-7c281b6c1af4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:22.984903 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c5c333-9a95-4772-9faa-7c281b6c1af4 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:22.985185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c5c333-9a95-4772-9faa-7c281b6c1af4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:22.985461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c5c333-9a95-4772-9faa-7c281b6c1af4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:22.986101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c5c333-9a95-4772-9faa-7c281b6c1af4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:22.987737 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c5c333-9a95-4772-9faa-7c281b6c1af4 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:22.999464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1094/2190] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:51:22 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:51:23.003695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c428bdb-a836-4b69-a689-e8ca34bd13cc admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:23.014370 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c428bdb-a836-4b69-a689-e8ca34bd13cc admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:23.019131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425f1fcb-e3e8-40c7-89c2-7b185e59c82e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:23.019787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425f1fcb-e3e8-40c7-89c2-7b185e59c82e admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:23.020066 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425f1fcb-e3e8-40c7-89c2-7b185e59c82e admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:23.020341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425f1fcb-e3e8-40c7-89c2-7b185e59c82e admin admin] PATH_INFO: `/v3/projects/62ca6d23922447b887ffda4bee7afd95/users/cb9100dde227460fb84077685997361c/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:23.036239 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-425f1fcb-e3e8-40c7-89c2-7b185e59c82e admin admin] RBAC: Authorizing `identity:create_grant(project_id=62ca6d23922447b887ffda4bee7afd95, user_id=cb9100dde227460fb84077685997361c, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:23.038842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-425f1fcb-e3e8-40c7-89c2-7b185e59c82e admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:23.129323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1097/2191] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:51:23 2022] PUT /identity/v3/projects/62ca6d23922447b887ffda4bee7afd95/users/cb9100dde227460fb84077685997361c/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:51:23.140898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e94c607-c571-475c-b162-bcb1effba3a7 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:23.141189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e94c607-c571-475c-b162-bcb1effba3a7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:23.141488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e94c607-c571-475c-b162-bcb1effba3a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:23.168489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-0e94c607-c571-475c-b162-bcb1effba3a7 None None] MFA Rules not processed for user `cb9100dde227460fb84077685997361c`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:51:23.169682 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0e94c607-c571-475c-b162-bcb1effba3a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:23.179197 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:51:23.236035 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1095/2192] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:51:23 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:51:23.246953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425f1fcb-e3e8-40c7-89c2-7b185e59c82e admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:23.249376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-425f1fcb-e3e8-40c7-89c2-7b185e59c82e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:23.314138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425f1fcb-e3e8-40c7-89c2-7b185e59c82e admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:23.318125 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249cd6c1-a97a-4309-8b8d-84f8c28394a5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:23.318741 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249cd6c1-a97a-4309-8b8d-84f8c28394a5 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:23.319040 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249cd6c1-a97a-4309-8b8d-84f8c28394a5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:23.319515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249cd6c1-a97a-4309-8b8d-84f8c28394a5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:23.331617 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249cd6c1-a97a-4309-8b8d-84f8c28394a5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:23.333316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249cd6c1-a97a-4309-8b8d-84f8c28394a5 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:23.347407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1098/2193] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:51:23 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:51:27.442546 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f33ae18-5cca-49ab-a012-1abd771953b5 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:27.443135 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f33ae18-5cca-49ab-a012-1abd771953b5 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:27.443952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f33ae18-5cca-49ab-a012-1abd771953b5 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:27.445840 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1096/2194] 10.222.0.49 () {54 vars in 921 bytes} [Fri Sep 16 03:51:27 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:51:27.452232 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35134d12-575a-45b3-957b-837219e45c53 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:27.452721 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35134d12-575a-45b3-957b-837219e45c53 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:27.452831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35134d12-575a-45b3-957b-837219e45c53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:27.493813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-35134d12-575a-45b3-957b-837219e45c53 None None] MFA Rules not processed for user `ded1bcd10e62499a9aaa0ece5aa55c80`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:51:27.498830 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-35134d12-575a-45b3-957b-837219e45c53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:27.589502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1099/2195] 10.222.0.49 () {58 vars in 1018 bytes} [Fri Sep 16 03:51:27 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:51:27.602111 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f33ae18-5cca-49ab-a012-1abd771953b5 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:27.604136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1f33ae18-5cca-49ab-a012-1abd771953b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:27.666350 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f33ae18-5cca-49ab-a012-1abd771953b5 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:27.670086 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23665d68-d154-470b-a448-9906b7fb715e service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ceb9fc31c345ccabaee8f30a3982a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:27.670689 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23665d68-d154-470b-a448-9906b7fb715e service glance] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:27.670992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23665d68-d154-470b-a448-9906b7fb715e service glance] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:27.671311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23665d68-d154-470b-a448-9906b7fb715e service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:27.682046 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23665d68-d154-470b-a448-9906b7fb715e service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:27.683492 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23665d68-d154-470b-a448-9906b7fb715e service glance] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:27.694733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1097/2196] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:51:27 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:51:27.963440 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adbaf46-6ca0-4e94-b881-9026067a7bcd None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:27.963855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adbaf46-6ca0-4e94-b881-9026067a7bcd None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:27.965676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adbaf46-6ca0-4e94-b881-9026067a7bcd None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:27.966218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1100/2197] 10.222.0.49 () {54 vars in 921 bytes} [Fri Sep 16 03:51:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:51:27.974435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047fafbd-27ec-472f-afb5-1ab3d92a7a04 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:27.974815 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047fafbd-27ec-472f-afb5-1ab3d92a7a04 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:27.975216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047fafbd-27ec-472f-afb5-1ab3d92a7a04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:27.997690 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-047fafbd-27ec-472f-afb5-1ab3d92a7a04 None None] MFA Rules not processed for user `f9275eeba471423e9fa2a31a6e08676d`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:51:27.999363 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-047fafbd-27ec-472f-afb5-1ab3d92a7a04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:28.051559 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1098/2198] 10.222.0.49 () {58 vars in 1018 bytes} [Fri Sep 16 03:51:27 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:51:28.065037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adbaf46-6ca0-4e94-b881-9026067a7bcd None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:28.066648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-6adbaf46-6ca0-4e94-b881-9026067a7bcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:28.083941 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adbaf46-6ca0-4e94-b881-9026067a7bcd None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:28.083941 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873bb9a0-0048-4367-9efa-47667475356b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:28.084214 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873bb9a0-0048-4367-9efa-47667475356b service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:28.084427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873bb9a0-0048-4367-9efa-47667475356b service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:28.084704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873bb9a0-0048-4367-9efa-47667475356b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:28.091953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873bb9a0-0048-4367-9efa-47667475356b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:28.093492 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873bb9a0-0048-4367-9efa-47667475356b service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:28.102473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1101/2199] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:51:28 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:51:28.222948 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9b90a6-eca9-452d-b56c-ef1a5d7611a4 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:28.223285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9b90a6-eca9-452d-b56c-ef1a5d7611a4 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:28.223567 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9b90a6-eca9-452d-b56c-ef1a5d7611a4 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:28.225045 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1099/2200] 10.222.0.49 () {54 vars in 921 bytes} [Fri Sep 16 03:51:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:51:28.230569 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d08ac6-71e7-4751-aa2f-947724fdb6c6 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:28.230934 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d08ac6-71e7-4751-aa2f-947724fdb6c6 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:28.231220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d08ac6-71e7-4751-aa2f-947724fdb6c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:28.252646 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-a7d08ac6-71e7-4751-aa2f-947724fdb6c6 None None] MFA Rules not processed for user `ded1bcd10e62499a9aaa0ece5aa55c80`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:51:28.253867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a7d08ac6-71e7-4751-aa2f-947724fdb6c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:28.263421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1102/2201] 10.222.0.49 () {58 vars in 1018 bytes} [Fri Sep 16 03:51:28 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:51:28.274279 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9b90a6-eca9-452d-b56c-ef1a5d7611a4 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:28.276098 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ea9b90a6-eca9-452d-b56c-ef1a5d7611a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:28.291061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9b90a6-eca9-452d-b56c-ef1a5d7611a4 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:28.295587 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3120887a-95db-4f23-ba30-1158b29ce579 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:28.296197 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3120887a-95db-4f23-ba30-1158b29ce579 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:28.296476 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3120887a-95db-4f23-ba30-1158b29ce579 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:28.296752 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3120887a-95db-4f23-ba30-1158b29ce579 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:28.312079 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3120887a-95db-4f23-ba30-1158b29ce579 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:28.312079 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3120887a-95db-4f23-ba30-1158b29ce579 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:28.321653 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1100/2202] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:51:28 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:51:33.057544 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c486ec-04d8-4505-9182-b358a018a3c8 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:33.058341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c486ec-04d8-4505-9182-b358a018a3c8 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:33.058341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c486ec-04d8-4505-9182-b358a018a3c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:33.066179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:51:33.083545 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-b5c486ec-04d8-4505-9182-b358a018a3c8 None None] MFA Rules not processed for user `d40810f61ead4c3da414194b325da80a`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:51:33.084761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b5c486ec-04d8-4505-9182-b358a018a3c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:33.157139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1103/2203] 10.222.0.49 () {58 vars in 1009 bytes} [Fri Sep 16 03:51:33 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:51:33.172921 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3120887a-95db-4f23-ba30-1158b29ce579 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:33.174801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3120887a-95db-4f23-ba30-1158b29ce579 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:33.233808 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:51:33.252574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3120887a-95db-4f23-ba30-1158b29ce579 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:33.258001 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697ff8af-6940-4b60-9172-42d7d3201b15 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa30da9972e140d7905c522576681429', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:33.258614 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697ff8af-6940-4b60-9172-42d7d3201b15 service swift] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:33.258888 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697ff8af-6940-4b60-9172-42d7d3201b15 service swift] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:33.259198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697ff8af-6940-4b60-9172-42d7d3201b15 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:33.269331 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697ff8af-6940-4b60-9172-42d7d3201b15 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:33.270968 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697ff8af-6940-4b60-9172-42d7d3201b15 service swift] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:33.281733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1101/2204] 10.222.0.49 () {60 vars in 1384 bytes} [Fri Sep 16 03:51:33 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 03:51:35.460492 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c486ec-04d8-4505-9182-b358a018a3c8 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:35.528250 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c486ec-04d8-4505-9182-b358a018a3c8 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:35.533012 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555a1a6c-79ea-4d19-93ae-e8fd962d8e61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:35.533658 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555a1a6c-79ea-4d19-93ae-e8fd962d8e61 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:35.533951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555a1a6c-79ea-4d19-93ae-e8fd962d8e61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:35.534226 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555a1a6c-79ea-4d19-93ae-e8fd962d8e61 admin admin] PATH_INFO: `/v3/users/cb9100dde227460fb84077685997361c` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:35.535415 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555a1a6c-79ea-4d19-93ae-e8fd962d8e61 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cb9100dde227460fb84077685997361c)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:35.537380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555a1a6c-79ea-4d19-93ae-e8fd962d8e61 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:36.254524 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-555a1a6c-79ea-4d19-93ae-e8fd962d8e61 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cb9100dde227460fb84077685997361c'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:36.318296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-555a1a6c-79ea-4d19-93ae-e8fd962d8e61 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cb9100dde227460fb84077685997361c'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:36.334534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-555a1a6c-79ea-4d19-93ae-e8fd962d8e61 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cb9100dde227460fb84077685997361c'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:36.335285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-555a1a6c-79ea-4d19-93ae-e8fd962d8e61 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cb9100dde227460fb84077685997361c'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:36.343637 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-555a1a6c-79ea-4d19-93ae-e8fd962d8e61 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cb9100dde227460fb84077685997361c'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:36.349497 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1104/2205] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:51:35 2022] DELETE /identity/v3/users/cb9100dde227460fb84077685997361c => generated 0 bytes in 889 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:51:36.354107 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44614013-be38-4a80-b6e6-77e1c1ec9ee6 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:36.354659 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44614013-be38-4a80-b6e6-77e1c1ec9ee6 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:36.354659 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44614013-be38-4a80-b6e6-77e1c1ec9ee6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:36.373986 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-44614013-be38-4a80-b6e6-77e1c1ec9ee6 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:51:36.375373 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-44614013-be38-4a80-b6e6-77e1c1ec9ee6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:36.449399 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1102/2206] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:51:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:51:36.460099 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555a1a6c-79ea-4d19-93ae-e8fd962d8e61 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:36.461632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-555a1a6c-79ea-4d19-93ae-e8fd962d8e61 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:36.496318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:51:36.523789 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555a1a6c-79ea-4d19-93ae-e8fd962d8e61 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:36.527416 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8118f6-6d87-4599-a0d8-b5e8847c4647 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:36.528021 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8118f6-6d87-4599-a0d8-b5e8847c4647 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:36.528374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8118f6-6d87-4599-a0d8-b5e8847c4647 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:36.528654 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8118f6-6d87-4599-a0d8-b5e8847c4647 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:36.538253 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8118f6-6d87-4599-a0d8-b5e8847c4647 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:36.539663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8118f6-6d87-4599-a0d8-b5e8847c4647 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:36.549186 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1105/2207] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:51:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:51:37.237556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44614013-be38-4a80-b6e6-77e1c1ec9ee6 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:51:37.239113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-44614013-be38-4a80-b6e6-77e1c1ec9ee6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:51:37.253427 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44614013-be38-4a80-b6e6-77e1c1ec9ee6 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:51:37.261019 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1e7829-4247-4db8-ac37-878c64ae37e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:51:37.261908 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1e7829-4247-4db8-ac37-878c64ae37e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:51:37.262265 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1e7829-4247-4db8-ac37-878c64ae37e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:51:37.262858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1e7829-4247-4db8-ac37-878c64ae37e3 admin admin] PATH_INFO: `/v3/projects/62ca6d23922447b887ffda4bee7afd95` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:51:37.264468 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1e7829-4247-4db8-ac37-878c64ae37e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=62ca6d23922447b887ffda4bee7afd95)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:51:37.266490 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1e7829-4247-4db8-ac37-878c64ae37e3 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:51:37.410070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6d1e7829-4247-4db8-ac37-878c64ae37e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '62ca6d23922447b887ffda4bee7afd95', 'request_id': 'req-6d1e7829-4247-4db8-ac37-878c64ae37e3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:37.416604 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6d1e7829-4247-4db8-ac37-878c64ae37e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '62ca6d23922447b887ffda4bee7afd95', 'request_id': 'req-6d1e7829-4247-4db8-ac37-878c64ae37e3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:37.484672 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6d1e7829-4247-4db8-ac37-878c64ae37e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '62ca6d23922447b887ffda4bee7afd95', 'request_id': 'req-6d1e7829-4247-4db8-ac37-878c64ae37e3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:37.496987 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6d1e7829-4247-4db8-ac37-878c64ae37e3 admin admin] The token cache is being invalidate because project 62ca6d23922447b887ffda4bee7afd95 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:51:37.497451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6d1e7829-4247-4db8-ac37-878c64ae37e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:37.498101 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6d1e7829-4247-4db8-ac37-878c64ae37e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:51:37.500178 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1103/2208] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:51:37 2022] DELETE /identity/v3/projects/62ca6d23922447b887ffda4bee7afd95 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:51:46.612117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=14 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:51:47.493432 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=63,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:52:09.282192 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa3941f-0791-45b5-b45d-44fa64cae94d None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:52:09.282733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa3941f-0791-45b5-b45d-44fa64cae94d None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:52:09.282863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa3941f-0791-45b5-b45d-44fa64cae94d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:52:09.303971 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-4fa3941f-0791-45b5-b45d-44fa64cae94d None None] MFA Rules not processed for user `d40810f61ead4c3da414194b325da80a`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:52:09.305441 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-4fa3941f-0791-45b5-b45d-44fa64cae94d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:52:09.367550 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1106/2209] 10.222.0.49 () {58 vars in 967 bytes} [Fri Sep 16 03:52:09 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:52:09.391042 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1e7829-4247-4db8-ac37-878c64ae37e3 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:52:09.393518 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6d1e7829-4247-4db8-ac37-878c64ae37e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:52:09.464838 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1e7829-4247-4db8-ac37-878c64ae37e3 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:52:09.469928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bb4165-5ac1-42cf-8b6a-d09b9b124dba service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa30da9972e140d7905c522576681429', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:52:09.470533 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bb4165-5ac1-42cf-8b6a-d09b9b124dba service swift] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:52:09.471350 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bb4165-5ac1-42cf-8b6a-d09b9b124dba service swift] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:52:09.473072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bb4165-5ac1-42cf-8b6a-d09b9b124dba service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:52:09.479835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bb4165-5ac1-42cf-8b6a-d09b9b124dba service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:52:09.481461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bb4165-5ac1-42cf-8b6a-d09b9b124dba service swift] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:52:09.490222 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1104/2210] 10.222.0.49 () {60 vars in 1384 bytes} [Fri Sep 16 03:52:09 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 03:52:19.359637 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=25 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:52:19.486347 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=32 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:52:31.771525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d689d88f-c310-42fa-9f53-79520017ec28 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:52:31.772097 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d689d88f-c310-42fa-9f53-79520017ec28 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:52:31.772207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d689d88f-c310-42fa-9f53-79520017ec28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:52:31.792786 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-d689d88f-c310-42fa-9f53-79520017ec28 None None] MFA Rules not processed for user `5691b0a2de144779b3b9a276f4f9aa79`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:52:31.793973 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d689d88f-c310-42fa-9f53-79520017ec28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:52:31.848186 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1107/2211] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:52:31 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:52:41.841002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=25 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:52:47.596269 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347764e8-8f4d-49de-9d0d-437e6c1ae61e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:52:47.597048 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347764e8-8f4d-49de-9d0d-437e6c1ae61e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:52:47.597048 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347764e8-8f4d-49de-9d0d-437e6c1ae61e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:52:47.617141 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-347764e8-8f4d-49de-9d0d-437e6c1ae61e None None] MFA Rules not processed for user `ff7df9be8475405eba924b59a7bd6bb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:52:47.621347 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-347764e8-8f4d-49de-9d0d-437e6c1ae61e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:52:47.674390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1105/2212] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:52:47 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:52:50.548171 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf833e4-256d-446e-8cc5-24842e065898 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:52:50.548812 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf833e4-256d-446e-8cc5-24842e065898 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:52:50.548812 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf833e4-256d-446e-8cc5-24842e065898 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:52:50.569175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-fbf833e4-256d-446e-8cc5-24842e065898 None None] MFA Rules not processed for user `a6b29b9086f64ea7a7990572567642f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:52:50.570688 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-fbf833e4-256d-446e-8cc5-24842e065898 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:52:50.630983 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1108/2213] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:52:50 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:52:57.668415 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=25 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:53:00.624069 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=25 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:53:11.908687 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347764e8-8f4d-49de-9d0d-437e6c1ae61e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:11.910112 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-347764e8-8f4d-49de-9d0d-437e6c1ae61e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:11.973256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347764e8-8f4d-49de-9d0d-437e6c1ae61e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:11.976945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e764854-13cf-4bd5-89ea-5b2619854a76 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:11.980858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e764854-13cf-4bd5-89ea-5b2619854a76 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:11.980858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e764854-13cf-4bd5-89ea-5b2619854a76 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:11.980858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e764854-13cf-4bd5-89ea-5b2619854a76 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:11.980858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0e764854-13cf-4bd5-89ea-5b2619854a76 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:12.061980 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e764854-13cf-4bd5-89ea-5b2619854a76 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:12.064046 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e764854-13cf-4bd5-89ea-5b2619854a76 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:12.075403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1106/2214] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:53:11 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:53:13.281843 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4375a783-9da2-4483-894a-76dd5e4ab7bd None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:13.282435 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4375a783-9da2-4483-894a-76dd5e4ab7bd None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:13.282535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4375a783-9da2-4483-894a-76dd5e4ab7bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:13.303635 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-4375a783-9da2-4483-894a-76dd5e4ab7bd None None] MFA Rules not processed for user `416479ba056f4612b6214cf115251269`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:53:13.304853 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-4375a783-9da2-4483-894a-76dd5e4ab7bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:13.367467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1109/2215] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:53:13 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:53:14.188729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e764854-13cf-4bd5-89ea-5b2619854a76 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:14.190136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0e764854-13cf-4bd5-89ea-5b2619854a76 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:14.200862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e764854-13cf-4bd5-89ea-5b2619854a76 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:14.205352 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe51db51-dcd6-4ba0-bee1-a4dbfb398d33 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:14.205957 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe51db51-dcd6-4ba0-bee1-a4dbfb398d33 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:14.206245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe51db51-dcd6-4ba0-bee1-a4dbfb398d33 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:14.206525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe51db51-dcd6-4ba0-bee1-a4dbfb398d33 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:14.208252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fe51db51-dcd6-4ba0-bee1-a4dbfb398d33 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:14.222303 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe51db51-dcd6-4ba0-bee1-a4dbfb398d33 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:14.223396 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe51db51-dcd6-4ba0-bee1-a4dbfb398d33 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:14.235034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1107/2216] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:53:14 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:53:23.360292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=25 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:53:24.230055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=60 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:53:25.077070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4375a783-9da2-4483-894a-76dd5e4ab7bd None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:25.079620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-4375a783-9da2-4483-894a-76dd5e4ab7bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:25.150535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4375a783-9da2-4483-894a-76dd5e4ab7bd None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:25.154645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3e56b3-6f5a-4a9a-b4ff-b3746be82ae5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:25.155631 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3e56b3-6f5a-4a9a-b4ff-b3746be82ae5 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:25.156764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3e56b3-6f5a-4a9a-b4ff-b3746be82ae5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:25.156764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3e56b3-6f5a-4a9a-b4ff-b3746be82ae5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:25.158842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1d3e56b3-6f5a-4a9a-b4ff-b3746be82ae5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:25.221043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3e56b3-6f5a-4a9a-b4ff-b3746be82ae5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:25.224073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3e56b3-6f5a-4a9a-b4ff-b3746be82ae5 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:25.233992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1110/2217] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:53:25 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:53:30.181880 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe51db51-dcd6-4ba0-bee1-a4dbfb398d33 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:30.183435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fe51db51-dcd6-4ba0-bee1-a4dbfb398d33 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:30.202083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe51db51-dcd6-4ba0-bee1-a4dbfb398d33 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:30.209038 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9ebb0c-7c19-40f0-afab-94032311320d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:30.209722 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9ebb0c-7c19-40f0-afab-94032311320d service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:30.210075 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9ebb0c-7c19-40f0-afab-94032311320d service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:30.210421 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9ebb0c-7c19-40f0-afab-94032311320d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:30.212287 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8e9ebb0c-7c19-40f0-afab-94032311320d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:30.223785 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9ebb0c-7c19-40f0-afab-94032311320d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:30.225460 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9ebb0c-7c19-40f0-afab-94032311320d service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:30.234714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1108/2218] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:53:30 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:53:35.229274 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=48 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:53:36.824224 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea9758a-7f99-4e2c-be33-1f81c4586a2d None None] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:36.824842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea9758a-7f99-4e2c-be33-1f81c4586a2d None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:36.824842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea9758a-7f99-4e2c-be33-1f81c4586a2d None None] PATH_INFO: `/` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:36.826374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1111/2219] 10.222.0.49 () {54 vars in 961 bytes} [Fri Sep 16 03:53:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 03:53:36.831953 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48f7097-5c72-40e6-846a-3e654b41b27d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:36.832448 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48f7097-5c72-40e6-846a-3e654b41b27d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:36.832552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48f7097-5c72-40e6-846a-3e654b41b27d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:36.852658 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-b48f7097-5c72-40e6-846a-3e654b41b27d None None] MFA Rules not processed for user `176da4679fcb46148da9289fccdad5c2`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:53:36.853655 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b48f7097-5c72-40e6-846a-3e654b41b27d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:36.928552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1109/2220] 10.222.0.49 () {58 vars in 1058 bytes} [Fri Sep 16 03:53:36 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:53:36.928738 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea9758a-7f99-4e2c-be33-1f81c4586a2d None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:36.949150 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea9758a-7f99-4e2c-be33-1f81c4586a2d None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:36.955675 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc67d9dd-0b1e-4469-98ae-062a677b6790 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '176da4679fcb46148da9289fccdad5c2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:36.956376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc67d9dd-0b1e-4469-98ae-062a677b6790 service placement] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:36.956725 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc67d9dd-0b1e-4469-98ae-062a677b6790 service placement] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:36.957063 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc67d9dd-0b1e-4469-98ae-062a677b6790 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:36.959124 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-cc67d9dd-0b1e-4469-98ae-062a677b6790 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:36.972110 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc67d9dd-0b1e-4469-98ae-062a677b6790 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:36.973691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc67d9dd-0b1e-4469-98ae-062a677b6790 service placement] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:36.984158 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1112/2221] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:53:36 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:53:38.403961 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48f7097-5c72-40e6-846a-3e654b41b27d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:38.405674 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b48f7097-5c72-40e6-846a-3e654b41b27d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:38.417550 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48f7097-5c72-40e6-846a-3e654b41b27d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:38.421374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8e06cd-d803-423a-9c30-ba2909d10cee service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:38.422001 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8e06cd-d803-423a-9c30-ba2909d10cee service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:38.422296 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8e06cd-d803-423a-9c30-ba2909d10cee service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:38.422572 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8e06cd-d803-423a-9c30-ba2909d10cee service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:38.424382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3f8e06cd-d803-423a-9c30-ba2909d10cee service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:38.437232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8e06cd-d803-423a-9c30-ba2909d10cee service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:38.438676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8e06cd-d803-423a-9c30-ba2909d10cee service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:38.449497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1110/2222] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:53:38 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:53:40.053275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7be4b8-6d5d-461b-b794-ce3194f1222a None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:40.053815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7be4b8-6d5d-461b-b794-ce3194f1222a None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:40.053815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7be4b8-6d5d-461b-b794-ce3194f1222a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:40.073200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-6a7be4b8-6d5d-461b-b794-ce3194f1222a None None] MFA Rules not processed for user `8f410d691e364880b13bb9144bbc5e7d`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:53:40.074438 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-6a7be4b8-6d5d-461b-b794-ce3194f1222a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:40.081377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1113/2223] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:53:40 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:53:40.437527 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8e06cd-d803-423a-9c30-ba2909d10cee service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:40.447859 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8e06cd-d803-423a-9c30-ba2909d10cee service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:40.451521 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4b88fe-2935-4804-9eea-045552524dac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:40.452119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4b88fe-2935-4804-9eea-045552524dac service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:40.452404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4b88fe-2935-4804-9eea-045552524dac service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:40.452676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4b88fe-2935-4804-9eea-045552524dac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:40.454389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-da4b88fe-2935-4804-9eea-045552524dac service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:40.464036 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4b88fe-2935-4804-9eea-045552524dac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:40.465467 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4b88fe-2935-4804-9eea-045552524dac service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:40.474541 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1111/2224] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:53:40 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:53:45.752692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7be4b8-6d5d-461b-b794-ce3194f1222a None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:45.755078 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-6a7be4b8-6d5d-461b-b794-ce3194f1222a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:45.835028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7be4b8-6d5d-461b-b794-ce3194f1222a None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:45.841412 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d668b2-e409-4ac1-adce-c53384564ac7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:45.842411 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d668b2-e409-4ac1-adce-c53384564ac7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:45.842924 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d668b2-e409-4ac1-adce-c53384564ac7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:45.843376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d668b2-e409-4ac1-adce-c53384564ac7 admin admin] PATH_INFO: `/v3/users/5691b0a2de144779b3b9a276f4f9aa79` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:45.845188 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d668b2-e409-4ac1-adce-c53384564ac7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5691b0a2de144779b3b9a276f4f9aa79)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:45.848303 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d668b2-e409-4ac1-adce-c53384564ac7 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:46.042966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-31d668b2-e409-4ac1-adce-c53384564ac7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5691b0a2de144779b3b9a276f4f9aa79'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:53:46.118636 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-31d668b2-e409-4ac1-adce-c53384564ac7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5691b0a2de144779b3b9a276f4f9aa79'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:53:46.134084 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-31d668b2-e409-4ac1-adce-c53384564ac7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5691b0a2de144779b3b9a276f4f9aa79'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:53:46.134791 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-31d668b2-e409-4ac1-adce-c53384564ac7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5691b0a2de144779b3b9a276f4f9aa79'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:53:46.141721 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-31d668b2-e409-4ac1-adce-c53384564ac7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5691b0a2de144779b3b9a276f4f9aa79'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:53:46.149214 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1114/2225] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:53:45 2022] DELETE /identity/v3/users/5691b0a2de144779b3b9a276f4f9aa79 => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:53:46.531723 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4b88fe-2935-4804-9eea-045552524dac service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:46.543922 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4b88fe-2935-4804-9eea-045552524dac service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:46.628415 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff5b619-e05b-4748-8005-8519a858c3ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:46.629575 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff5b619-e05b-4748-8005-8519a858c3ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:46.630232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff5b619-e05b-4748-8005-8519a858c3ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:46.630936 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff5b619-e05b-4748-8005-8519a858c3ea admin admin] PATH_INFO: `/v3/projects/0ea684aa585e4b4483261a014d861cb1` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:46.633496 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff5b619-e05b-4748-8005-8519a858c3ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ea684aa585e4b4483261a014d861cb1)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:46.636042 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff5b619-e05b-4748-8005-8519a858c3ea admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:46.831978 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eff5b619-e05b-4748-8005-8519a858c3ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ea684aa585e4b4483261a014d861cb1', 'request_id': 'req-eff5b619-e05b-4748-8005-8519a858c3ea'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:53:46.839864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eff5b619-e05b-4748-8005-8519a858c3ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ea684aa585e4b4483261a014d861cb1', 'request_id': 'req-eff5b619-e05b-4748-8005-8519a858c3ea'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:53:46.844394 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:53:46.907401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eff5b619-e05b-4748-8005-8519a858c3ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ea684aa585e4b4483261a014d861cb1', 'request_id': 'req-eff5b619-e05b-4748-8005-8519a858c3ea'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:53:46.920744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eff5b619-e05b-4748-8005-8519a858c3ea admin admin] The token cache is being invalidate because project 0ea684aa585e4b4483261a014d861cb1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:53:46.921074 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eff5b619-e05b-4748-8005-8519a858c3ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:53:46.921721 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eff5b619-e05b-4748-8005-8519a858c3ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:53:46.923440 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1112/2226] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:53:46 2022] DELETE /identity/v3/projects/0ea684aa585e4b4483261a014d861cb1 => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:53:46.928463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bf00e8-f94e-4d88-8824-59198bef9192 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:46.928950 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bf00e8-f94e-4d88-8824-59198bef9192 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:46.929068 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bf00e8-f94e-4d88-8824-59198bef9192 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:46.950661 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-f7bf00e8-f94e-4d88-8824-59198bef9192 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:53:46.951894 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-f7bf00e8-f94e-4d88-8824-59198bef9192 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:47.012251 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:53:47.027836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1115/2227] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:53:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:53:47.030977 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5e9637-ba8f-405a-bd2e-51d6f4a3a8f6 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:47.031266 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5e9637-ba8f-405a-bd2e-51d6f4a3a8f6 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:47.031541 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5e9637-ba8f-405a-bd2e-51d6f4a3a8f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:47.054774 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9a5e9637-ba8f-405a-bd2e-51d6f4a3a8f6 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:53:47.054774 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9a5e9637-ba8f-405a-bd2e-51d6f4a3a8f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:47.061458 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1113/2228] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:53:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:53:47.085184 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bf00e8-f94e-4d88-8824-59198bef9192 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:47.095844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bf00e8-f94e-4d88-8824-59198bef9192 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:47.100796 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55601dd5-030a-4ec2-81d7-0fd564f7b66d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:47.101410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55601dd5-030a-4ec2-81d7-0fd564f7b66d admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:47.101692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55601dd5-030a-4ec2-81d7-0fd564f7b66d admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:47.101993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55601dd5-030a-4ec2-81d7-0fd564f7b66d admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:47.102674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55601dd5-030a-4ec2-81d7-0fd564f7b66d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:53:47.103206 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55601dd5-030a-4ec2-81d7-0fd564f7b66d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:47.105008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55601dd5-030a-4ec2-81d7-0fd564f7b66d admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:47.117228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1116/2229] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:53:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:53:47.121860 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5e9637-ba8f-405a-bd2e-51d6f4a3a8f6 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:47.131909 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5e9637-ba8f-405a-bd2e-51d6f4a3a8f6 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:47.137281 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e313c378-67d1-4da7-b065-60662be450c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:47.138052 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e313c378-67d1-4da7-b065-60662be450c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:47.138499 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e313c378-67d1-4da7-b065-60662be450c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:47.138862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e313c378-67d1-4da7-b065-60662be450c5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:47.139636 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e313c378-67d1-4da7-b065-60662be450c5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:47.141646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e313c378-67d1-4da7-b065-60662be450c5 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:47.222790 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1114/2230] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:53:47 2022] POST /identity/v3/projects => generated 399 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:53:47.227402 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55601dd5-030a-4ec2-81d7-0fd564f7b66d admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:47.237439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55601dd5-030a-4ec2-81d7-0fd564f7b66d admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:47.299445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a7d345-0aad-4395-8edd-ebd5161cd889 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:47.300355 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a7d345-0aad-4395-8edd-ebd5161cd889 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:47.301506 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a7d345-0aad-4395-8edd-ebd5161cd889 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:47.301830 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a7d345-0aad-4395-8edd-ebd5161cd889 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:47.302570 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a7d345-0aad-4395-8edd-ebd5161cd889 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:47.304635 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a7d345-0aad-4395-8edd-ebd5161cd889 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:47.399135 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1117/2231] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:53:47 2022] POST /identity/v3/users => generated 300 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:53:47.404290 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e313c378-67d1-4da7-b065-60662be450c5 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:47.414315 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e313c378-67d1-4da7-b065-60662be450c5 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:47.420094 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd51668-4922-4994-833c-ac67e78e8d29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:47.420713 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd51668-4922-4994-833c-ac67e78e8d29 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:47.420992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd51668-4922-4994-833c-ac67e78e8d29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:47.421268 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd51668-4922-4994-833c-ac67e78e8d29 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:47.422275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd51668-4922-4994-833c-ac67e78e8d29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:47.423702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd51668-4922-4994-833c-ac67e78e8d29 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:47.441127 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1115/2232] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:53:47 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:53:47.442431 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a7d345-0aad-4395-8edd-ebd5161cd889 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:47.457202 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a7d345-0aad-4395-8edd-ebd5161cd889 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:47.464026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11688ee6-f731-4fc3-a094-265a7c5110ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:47.464899 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11688ee6-f731-4fc3-a094-265a7c5110ca admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:47.465301 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11688ee6-f731-4fc3-a094-265a7c5110ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:47.465581 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11688ee6-f731-4fc3-a094-265a7c5110ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:47.466227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11688ee6-f731-4fc3-a094-265a7c5110ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:47.467839 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11688ee6-f731-4fc3-a094-265a7c5110ca admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:47.481171 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1118/2233] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:53:47 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:53:47.484470 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd51668-4922-4994-833c-ac67e78e8d29 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:47.504756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd51668-4922-4994-833c-ac67e78e8d29 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:47.510508 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7407a5-6279-4b2d-928e-ee34662fc2c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:47.511331 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7407a5-6279-4b2d-928e-ee34662fc2c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:47.511666 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7407a5-6279-4b2d-928e-ee34662fc2c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:47.512008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7407a5-6279-4b2d-928e-ee34662fc2c3 admin admin] PATH_INFO: `/v3/projects/43f04e4d15c34e70bacc36db4ed53557/users/de8a489911b64f49ac7ff85597f2e413/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:47.530609 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7407a5-6279-4b2d-928e-ee34662fc2c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=43f04e4d15c34e70bacc36db4ed53557, user_id=de8a489911b64f49ac7ff85597f2e413, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:47.533441 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7407a5-6279-4b2d-928e-ee34662fc2c3 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:47.615358 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1116/2234] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:53:47 2022] PUT /identity/v3/projects/43f04e4d15c34e70bacc36db4ed53557/users/de8a489911b64f49ac7ff85597f2e413/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:53:47.626166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff613e4d-eb11-4cdf-9338-30c89e2a6bd0 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:47.638879 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff613e4d-eb11-4cdf-9338-30c89e2a6bd0 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:47.638879 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff613e4d-eb11-4cdf-9338-30c89e2a6bd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:47.675517 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-ff613e4d-eb11-4cdf-9338-30c89e2a6bd0 None None] MFA Rules not processed for user `de8a489911b64f49ac7ff85597f2e413`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:53:47.688241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ff613e4d-eb11-4cdf-9338-30c89e2a6bd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:47.792161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1119/2235] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:53:47 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:53:47.806929 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7407a5-6279-4b2d-928e-ee34662fc2c3 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:47.806929 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4e7407a5-6279-4b2d-928e-ee34662fc2c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:47.877141 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7407a5-6279-4b2d-928e-ee34662fc2c3 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:47.881679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e92b4e0-86bd-42f8-93f4-de2d8313820e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:47.884383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e92b4e0-86bd-42f8-93f4-de2d8313820e service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:47.884647 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e92b4e0-86bd-42f8-93f4-de2d8313820e service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:47.884931 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e92b4e0-86bd-42f8-93f4-de2d8313820e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:47.895714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e92b4e0-86bd-42f8-93f4-de2d8313820e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:47.897225 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e92b4e0-86bd-42f8-93f4-de2d8313820e service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:47.908025 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1117/2236] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:53:47 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:53:48.511779 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff613e4d-eb11-4cdf-9338-30c89e2a6bd0 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:48.513193 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ff613e4d-eb11-4cdf-9338-30c89e2a6bd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:48.586582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff613e4d-eb11-4cdf-9338-30c89e2a6bd0 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:48.590294 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b9ea9e-1066-437a-954b-669943dc3a08 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:48.590975 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b9ea9e-1066-437a-954b-669943dc3a08 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:48.591266 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b9ea9e-1066-437a-954b-669943dc3a08 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:48.591546 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b9ea9e-1066-437a-954b-669943dc3a08 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:48.593360 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-21b9ea9e-1066-437a-954b-669943dc3a08 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:48.605685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b9ea9e-1066-437a-954b-669943dc3a08 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:48.611693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b9ea9e-1066-437a-954b-669943dc3a08 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:48.611693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:53:48.617756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1120/2237] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:53:48 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:53:49.665214 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e92b4e0-86bd-42f8-93f4-de2d8313820e service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:49.666803 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7e92b4e0-86bd-42f8-93f4-de2d8313820e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:49.767786 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e92b4e0-86bd-42f8-93f4-de2d8313820e service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:49.775458 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0ea4ad-9d82-47a2-9909-48c0e08428c3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:49.776127 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0ea4ad-9d82-47a2-9909-48c0e08428c3 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:49.776517 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0ea4ad-9d82-47a2-9909-48c0e08428c3 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:49.776772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0ea4ad-9d82-47a2-9909-48c0e08428c3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:49.778895 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ba0ea4ad-9d82-47a2-9909-48c0e08428c3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:49.792540 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0ea4ad-9d82-47a2-9909-48c0e08428c3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:49.794480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0ea4ad-9d82-47a2-9909-48c0e08428c3 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:49.807323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1118/2238] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:53:49 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:53:53.838518 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b9ea9e-1066-437a-954b-669943dc3a08 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:53.842691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-21b9ea9e-1066-437a-954b-669943dc3a08 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:53.856431 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b9ea9e-1066-437a-954b-669943dc3a08 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:53.862089 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f9072b-aa3d-4eee-8c54-f5566336d254 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:53.862687 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f9072b-aa3d-4eee-8c54-f5566336d254 service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:53.863004 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f9072b-aa3d-4eee-8c54-f5566336d254 service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:53.863320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f9072b-aa3d-4eee-8c54-f5566336d254 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:53.865431 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-05f9072b-aa3d-4eee-8c54-f5566336d254 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:53.931043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f9072b-aa3d-4eee-8c54-f5566336d254 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:53.934508 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f9072b-aa3d-4eee-8c54-f5566336d254 service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:53.944240 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1121/2239] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:53:53 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:53:57.291027 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0ea4ad-9d82-47a2-9909-48c0e08428c3 service nova] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:57.303431 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:53:57.355538 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0ea4ad-9d82-47a2-9909-48c0e08428c3 service nova] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:57.360046 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce06dee-4e38-43e8-912a-0260686ea0b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:57.360664 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce06dee-4e38-43e8-912a-0260686ea0b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:57.360942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce06dee-4e38-43e8-912a-0260686ea0b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:57.361211 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce06dee-4e38-43e8-912a-0260686ea0b3 admin admin] PATH_INFO: `/v3/users/de8a489911b64f49ac7ff85597f2e413` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:57.362317 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce06dee-4e38-43e8-912a-0260686ea0b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=de8a489911b64f49ac7ff85597f2e413)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:57.365451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce06dee-4e38-43e8-912a-0260686ea0b3 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:57.555579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6ce06dee-4e38-43e8-912a-0260686ea0b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de8a489911b64f49ac7ff85597f2e413'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:53:57.572946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6ce06dee-4e38-43e8-912a-0260686ea0b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de8a489911b64f49ac7ff85597f2e413'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:53:57.667030 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6ce06dee-4e38-43e8-912a-0260686ea0b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de8a489911b64f49ac7ff85597f2e413'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:53:57.667736 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6ce06dee-4e38-43e8-912a-0260686ea0b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de8a489911b64f49ac7ff85597f2e413'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:53:57.676212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6ce06dee-4e38-43e8-912a-0260686ea0b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de8a489911b64f49ac7ff85597f2e413'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:53:57.683031 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1119/2240] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:53:57 2022] DELETE /identity/v3/users/de8a489911b64f49ac7ff85597f2e413 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:53:57.688054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabe21be-3ec5-4e2d-abec-56fb82570185 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:57.688054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabe21be-3ec5-4e2d-abec-56fb82570185 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:57.688802 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabe21be-3ec5-4e2d-abec-56fb82570185 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:57.709793 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-dabe21be-3ec5-4e2d-abec-56fb82570185 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:53:57.712030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-dabe21be-3ec5-4e2d-abec-56fb82570185 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:57.788010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1122/2241] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:53:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:53:57.797660 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce06dee-4e38-43e8-912a-0260686ea0b3 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:57.799216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6ce06dee-4e38-43e8-912a-0260686ea0b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:57.860309 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce06dee-4e38-43e8-912a-0260686ea0b3 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:57.864859 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b907d7a-e16d-462f-958a-22712edebdb4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:57.865521 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b907d7a-e16d-462f-958a-22712edebdb4 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:57.865802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b907d7a-e16d-462f-958a-22712edebdb4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:57.866105 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b907d7a-e16d-462f-958a-22712edebdb4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:57.875253 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b907d7a-e16d-462f-958a-22712edebdb4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:57.876729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b907d7a-e16d-462f-958a-22712edebdb4 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:57.891547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1120/2242] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:53:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:53:58.452114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabe21be-3ec5-4e2d-abec-56fb82570185 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:58.453946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-dabe21be-3ec5-4e2d-abec-56fb82570185 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:58.476368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabe21be-3ec5-4e2d-abec-56fb82570185 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:58.486787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104c66db-4853-476a-8cb6-a3c3a67c2351 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:58.487792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104c66db-4853-476a-8cb6-a3c3a67c2351 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:58.488016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104c66db-4853-476a-8cb6-a3c3a67c2351 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:58.488872 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104c66db-4853-476a-8cb6-a3c3a67c2351 admin admin] PATH_INFO: `/v3/projects/43f04e4d15c34e70bacc36db4ed53557` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:58.490825 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104c66db-4853-476a-8cb6-a3c3a67c2351 admin admin] RBAC: Authorizing `identity:delete_project(project_id=43f04e4d15c34e70bacc36db4ed53557)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:58.493005 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104c66db-4853-476a-8cb6-a3c3a67c2351 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:58.652709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-104c66db-4853-476a-8cb6-a3c3a67c2351 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '43f04e4d15c34e70bacc36db4ed53557', 'request_id': 'req-104c66db-4853-476a-8cb6-a3c3a67c2351'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:53:58.736061 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-104c66db-4853-476a-8cb6-a3c3a67c2351 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '43f04e4d15c34e70bacc36db4ed53557', 'request_id': 'req-104c66db-4853-476a-8cb6-a3c3a67c2351'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:53:58.744118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-104c66db-4853-476a-8cb6-a3c3a67c2351 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '43f04e4d15c34e70bacc36db4ed53557', 'request_id': 'req-104c66db-4853-476a-8cb6-a3c3a67c2351'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:53:58.757714 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-104c66db-4853-476a-8cb6-a3c3a67c2351 admin admin] The token cache is being invalidate because project 43f04e4d15c34e70bacc36db4ed53557 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:53:58.758209 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-104c66db-4853-476a-8cb6-a3c3a67c2351 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:53:58.758894 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-104c66db-4853-476a-8cb6-a3c3a67c2351 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:53:58.764472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1123/2243] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:53:58 2022] DELETE /identity/v3/projects/43f04e4d15c34e70bacc36db4ed53557 => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:53:58.767048 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260610b1-453f-4e81-8094-edf17a1f0962 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:58.767551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260610b1-453f-4e81-8094-edf17a1f0962 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:58.767652 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260610b1-453f-4e81-8094-edf17a1f0962 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:58.804292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-260610b1-453f-4e81-8094-edf17a1f0962 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:53:58.805501 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-260610b1-453f-4e81-8094-edf17a1f0962 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:58.837724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:53:58.870154 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1121/2244] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:53:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:53:58.883487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4985ac-3463-4680-96e3-b5b988c58d59 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:58.883864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4985ac-3463-4680-96e3-b5b988c58d59 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:58.884224 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4985ac-3463-4680-96e3-b5b988c58d59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:58.926861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-2a4985ac-3463-4680-96e3-b5b988c58d59 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:53:58.928788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-2a4985ac-3463-4680-96e3-b5b988c58d59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:58.955805 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1124/2245] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:53:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:53:58.955904 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260610b1-453f-4e81-8094-edf17a1f0962 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:58.963615 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260610b1-453f-4e81-8094-edf17a1f0962 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:58.968789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cadf9e-c97e-4cd5-96a8-36b58d122757 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:58.969402 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cadf9e-c97e-4cd5-96a8-36b58d122757 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:58.969678 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cadf9e-c97e-4cd5-96a8-36b58d122757 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:58.969953 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cadf9e-c97e-4cd5-96a8-36b58d122757 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:58.971312 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cadf9e-c97e-4cd5-96a8-36b58d122757 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:53:58.971617 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cadf9e-c97e-4cd5-96a8-36b58d122757 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:58.973798 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cadf9e-c97e-4cd5-96a8-36b58d122757 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:59.001444 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1122/2246] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:53:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:53:59.007218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4985ac-3463-4680-96e3-b5b988c58d59 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:59.008041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cadf9e-c97e-4cd5-96a8-36b58d122757 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:59.009063 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-2a4985ac-3463-4680-96e3-b5b988c58d59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:59.019610 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cadf9e-c97e-4cd5-96a8-36b58d122757 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:59.025374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c102d2-188c-4134-84d4-f289f5f5f95f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:59.026455 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c102d2-188c-4134-84d4-f289f5f5f95f admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:59.026845 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c102d2-188c-4134-84d4-f289f5f5f95f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:59.030032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c102d2-188c-4134-84d4-f289f5f5f95f admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:59.030032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c102d2-188c-4134-84d4-f289f5f5f95f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:59.030254 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c102d2-188c-4134-84d4-f289f5f5f95f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:59.033838 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:53:59.122895 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4985ac-3463-4680-96e3-b5b988c58d59 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:59.129302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac7c8eb-dd1f-4dd2-8a14-d97f962c671c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:59.130149 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac7c8eb-dd1f-4dd2-8a14-d97f962c671c service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:59.130403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac7c8eb-dd1f-4dd2-8a14-d97f962c671c service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:59.130678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac7c8eb-dd1f-4dd2-8a14-d97f962c671c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:59.132599 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1ac7c8eb-dd1f-4dd2-8a14-d97f962c671c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:59.145486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac7c8eb-dd1f-4dd2-8a14-d97f962c671c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:59.147157 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac7c8eb-dd1f-4dd2-8a14-d97f962c671c service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:59.157738 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1125/2247] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:53:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:53:59.205836 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1123/2248] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:53:59 2022] POST /identity/v3/projects => generated 383 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:53:59.210642 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac7c8eb-dd1f-4dd2-8a14-d97f962c671c service nova] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:59.222175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac7c8eb-dd1f-4dd2-8a14-d97f962c671c service nova] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:59.283481 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eac289b-53a5-4882-804c-28b93f4d6faf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:59.284157 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eac289b-53a5-4882-804c-28b93f4d6faf admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:59.284442 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eac289b-53a5-4882-804c-28b93f4d6faf admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:59.284714 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eac289b-53a5-4882-804c-28b93f4d6faf admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:59.285491 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eac289b-53a5-4882-804c-28b93f4d6faf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:59.287519 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eac289b-53a5-4882-804c-28b93f4d6faf admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:59.484175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1126/2249] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:53:59 2022] POST /identity/v3/users => generated 292 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:53:59.489420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c102d2-188c-4134-84d4-f289f5f5f95f admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:59.499652 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c102d2-188c-4134-84d4-f289f5f5f95f admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:59.505439 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbb4945-cfef-40e6-9e41-ab8476d22e89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:59.506040 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbb4945-cfef-40e6-9e41-ab8476d22e89 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:59.506322 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbb4945-cfef-40e6-9e41-ab8476d22e89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:59.506603 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbb4945-cfef-40e6-9e41-ab8476d22e89 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:59.507271 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbb4945-cfef-40e6-9e41-ab8476d22e89 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:59.508846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbb4945-cfef-40e6-9e41-ab8476d22e89 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:59.519112 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1124/2250] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:53:59 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:53:59.523437 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eac289b-53a5-4882-804c-28b93f4d6faf admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:59.533308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eac289b-53a5-4882-804c-28b93f4d6faf admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:59.538569 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1b49b9-160c-4fc5-85ce-017c22d1cc2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:59.539213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1b49b9-160c-4fc5-85ce-017c22d1cc2f admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:59.539496 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1b49b9-160c-4fc5-85ce-017c22d1cc2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:59.539772 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1b49b9-160c-4fc5-85ce-017c22d1cc2f admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:59.540421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1b49b9-160c-4fc5-85ce-017c22d1cc2f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:59.542014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1b49b9-160c-4fc5-85ce-017c22d1cc2f admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:59.553361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1127/2251] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:53:59 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:53:59.557527 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbb4945-cfef-40e6-9e41-ab8476d22e89 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:59.567930 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbb4945-cfef-40e6-9e41-ab8476d22e89 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:59.572564 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9962f15-1848-4fb5-9290-e44d44439d60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:59.573567 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9962f15-1848-4fb5-9290-e44d44439d60 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:59.573567 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9962f15-1848-4fb5-9290-e44d44439d60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:59.573835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9962f15-1848-4fb5-9290-e44d44439d60 admin admin] PATH_INFO: `/v3/projects/c705c2f92c1646dba9c11aa6000166bc/users/269dbbd6d926496d8c5c704832de5763/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:59.588921 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9962f15-1848-4fb5-9290-e44d44439d60 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c705c2f92c1646dba9c11aa6000166bc, user_id=269dbbd6d926496d8c5c704832de5763, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:59.591697 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9962f15-1848-4fb5-9290-e44d44439d60 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:59.662579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1125/2252] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:53:59 2022] PUT /identity/v3/projects/c705c2f92c1646dba9c11aa6000166bc/users/269dbbd6d926496d8c5c704832de5763/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:53:59.670252 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ab8d65-0b68-48e8-aefb-96efa067e057 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:59.670549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ab8d65-0b68-48e8-aefb-96efa067e057 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:59.670822 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ab8d65-0b68-48e8-aefb-96efa067e057 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:59.693385 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-d6ab8d65-0b68-48e8-aefb-96efa067e057 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:53:59.694697 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d6ab8d65-0b68-48e8-aefb-96efa067e057 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:59.757829 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1128/2253] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:53:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:53:59.767401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9962f15-1848-4fb5-9290-e44d44439d60 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:53:59.770110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e9962f15-1848-4fb5-9290-e44d44439d60 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:53:59.835324 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9962f15-1848-4fb5-9290-e44d44439d60 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:53:59.839976 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a94704a-40c6-4fe3-b491-19603d10b955 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:53:59.840592 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a94704a-40c6-4fe3-b491-19603d10b955 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:53:59.840876 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a94704a-40c6-4fe3-b491-19603d10b955 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:53:59.841151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a94704a-40c6-4fe3-b491-19603d10b955 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:53:59.851552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a94704a-40c6-4fe3-b491-19603d10b955 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:53:59.852990 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a94704a-40c6-4fe3-b491-19603d10b955 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:53:59.864221 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1126/2254] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:53:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:54:07.097084 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ab8d65-0b68-48e8-aefb-96efa067e057 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:07.100881 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d6ab8d65-0b68-48e8-aefb-96efa067e057 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:07.141940 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ab8d65-0b68-48e8-aefb-96efa067e057 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:07.164797 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2f9596-f134-423e-928f-d78c8fc5c894 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:07.165697 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2f9596-f134-423e-928f-d78c8fc5c894 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:07.166170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2f9596-f134-423e-928f-d78c8fc5c894 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:07.166650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2f9596-f134-423e-928f-d78c8fc5c894 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:07.169733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-4e2f9596-f134-423e-928f-d78c8fc5c894 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:07.199306 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2f9596-f134-423e-928f-d78c8fc5c894 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:07.199306 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2f9596-f134-423e-928f-d78c8fc5c894 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:07.203046 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a346cb-c658-46e4-bb5d-80ab7ee8ec2d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:07.203532 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a346cb-c658-46e4-bb5d-80ab7ee8ec2d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:07.203695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a346cb-c658-46e4-bb5d-80ab7ee8ec2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:07.211822 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1129/2255] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:54:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:54:07.258214 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e0a346cb-c658-46e4-bb5d-80ab7ee8ec2d None None] MFA Rules not processed for user `269dbbd6d926496d8c5c704832de5763`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:54:07.259639 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e0a346cb-c658-46e4-bb5d-80ab7ee8ec2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:07.333431 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:54:07.347174 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1127/2256] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:54:07 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:54:07.350004 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2f9596-f134-423e-928f-d78c8fc5c894 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:07.356303 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:54:07.360482 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2f9596-f134-423e-928f-d78c8fc5c894 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:07.368303 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfca5e6-a5b1-4ac0-aae0-9d49f9425b78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:07.374148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfca5e6-a5b1-4ac0-aae0-9d49f9425b78 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:07.374148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfca5e6-a5b1-4ac0-aae0-9d49f9425b78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:07.374148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfca5e6-a5b1-4ac0-aae0-9d49f9425b78 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:07.374148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfca5e6-a5b1-4ac0-aae0-9d49f9425b78 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:07.374148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfca5e6-a5b1-4ac0-aae0-9d49f9425b78 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:07.508987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1130/2257] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:54:07 2022] POST /identity/v3/projects => generated 383 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:54:07.512902 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a346cb-c658-46e4-bb5d-80ab7ee8ec2d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:07.524616 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a346cb-c658-46e4-bb5d-80ab7ee8ec2d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:07.593502 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61d1aba-7614-4059-9a67-08b1dce7d706 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:07.594136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61d1aba-7614-4059-9a67-08b1dce7d706 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:07.594415 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61d1aba-7614-4059-9a67-08b1dce7d706 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:07.594688 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61d1aba-7614-4059-9a67-08b1dce7d706 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:07.595409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61d1aba-7614-4059-9a67-08b1dce7d706 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:07.598204 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61d1aba-7614-4059-9a67-08b1dce7d706 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:07.682228 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1128/2258] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:54:07 2022] POST /identity/v3/users => generated 298 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:54:07.686094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfca5e6-a5b1-4ac0-aae0-9d49f9425b78 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:07.700275 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfca5e6-a5b1-4ac0-aae0-9d49f9425b78 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:07.708966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca419ebf-ead3-4dc4-892d-50d276592c74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:07.709610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca419ebf-ead3-4dc4-892d-50d276592c74 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:07.709883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca419ebf-ead3-4dc4-892d-50d276592c74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:07.710153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca419ebf-ead3-4dc4-892d-50d276592c74 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:07.710797 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca419ebf-ead3-4dc4-892d-50d276592c74 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:07.712642 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca419ebf-ead3-4dc4-892d-50d276592c74 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:07.724288 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1131/2259] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:54:07 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:54:07.728922 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61d1aba-7614-4059-9a67-08b1dce7d706 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:07.743595 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61d1aba-7614-4059-9a67-08b1dce7d706 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:07.749175 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227cffc1-f9dc-4fc4-8fff-b85ac3b40306 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:07.750067 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227cffc1-f9dc-4fc4-8fff-b85ac3b40306 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:07.750506 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227cffc1-f9dc-4fc4-8fff-b85ac3b40306 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:07.751107 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227cffc1-f9dc-4fc4-8fff-b85ac3b40306 admin admin] PATH_INFO: `/v3/projects/0ff078cd8282494eb7dd5ff34d7bddcb/users/d21a2934705941ceb91e2f33a4024be1/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:07.770720 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227cffc1-f9dc-4fc4-8fff-b85ac3b40306 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ff078cd8282494eb7dd5ff34d7bddcb, user_id=d21a2934705941ceb91e2f33a4024be1, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:07.773543 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227cffc1-f9dc-4fc4-8fff-b85ac3b40306 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:07.857194 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1129/2260] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:54:07 2022] PUT /identity/v3/projects/0ff078cd8282494eb7dd5ff34d7bddcb/users/d21a2934705941ceb91e2f33a4024be1/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:54:07.861177 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca419ebf-ead3-4dc4-892d-50d276592c74 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:07.872639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca419ebf-ead3-4dc4-892d-50d276592c74 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:07.940212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68523a82-ab1b-4989-8292-2823807efdf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:07.940838 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68523a82-ab1b-4989-8292-2823807efdf4 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:07.941728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68523a82-ab1b-4989-8292-2823807efdf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:07.942178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68523a82-ab1b-4989-8292-2823807efdf4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:07.943115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68523a82-ab1b-4989-8292-2823807efdf4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:07.944730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68523a82-ab1b-4989-8292-2823807efdf4 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:07.955920 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1132/2261] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:54:07 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:54:07.960733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227cffc1-f9dc-4fc4-8fff-b85ac3b40306 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:07.972098 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227cffc1-f9dc-4fc4-8fff-b85ac3b40306 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:07.978184 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5190ce6-d134-4125-a557-7e4dc9a79ea6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:07.979170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5190ce6-d134-4125-a557-7e4dc9a79ea6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:07.982758 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5190ce6-d134-4125-a557-7e4dc9a79ea6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:07.982758 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5190ce6-d134-4125-a557-7e4dc9a79ea6 admin admin] PATH_INFO: `/v3/projects/0ff078cd8282494eb7dd5ff34d7bddcb/users/d21a2934705941ceb91e2f33a4024be1/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:07.982758 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5190ce6-d134-4125-a557-7e4dc9a79ea6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ff078cd8282494eb7dd5ff34d7bddcb, user_id=d21a2934705941ceb91e2f33a4024be1, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:07.985559 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5190ce6-d134-4125-a557-7e4dc9a79ea6 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:08.005728 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1130/2262] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:54:07 2022] PUT /identity/v3/projects/0ff078cd8282494eb7dd5ff34d7bddcb/users/d21a2934705941ceb91e2f33a4024be1/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:54:12.011324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68523a82-ab1b-4989-8292-2823807efdf4 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:12.013695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-68523a82-ab1b-4989-8292-2823807efdf4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:12.029118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68523a82-ab1b-4989-8292-2823807efdf4 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:12.096891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88a49f9-6bc2-45e7-a0d5-75f99d1ee3d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:12.097564 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88a49f9-6bc2-45e7-a0d5-75f99d1ee3d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:12.097842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88a49f9-6bc2-45e7-a0d5-75f99d1ee3d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:12.098116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88a49f9-6bc2-45e7-a0d5-75f99d1ee3d8 admin admin] PATH_INFO: `/v3/users/a6b29b9086f64ea7a7990572567642f7` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:12.099280 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88a49f9-6bc2-45e7-a0d5-75f99d1ee3d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a6b29b9086f64ea7a7990572567642f7)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:12.101342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88a49f9-6bc2-45e7-a0d5-75f99d1ee3d8 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:12.286571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d88a49f9-6bc2-45e7-a0d5-75f99d1ee3d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a6b29b9086f64ea7a7990572567642f7'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:54:12.369771 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d88a49f9-6bc2-45e7-a0d5-75f99d1ee3d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a6b29b9086f64ea7a7990572567642f7'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:54:12.385580 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:54:12.391513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d88a49f9-6bc2-45e7-a0d5-75f99d1ee3d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a6b29b9086f64ea7a7990572567642f7'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:54:12.392502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d88a49f9-6bc2-45e7-a0d5-75f99d1ee3d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a6b29b9086f64ea7a7990572567642f7'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:54:12.404765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d88a49f9-6bc2-45e7-a0d5-75f99d1ee3d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a6b29b9086f64ea7a7990572567642f7'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:54:12.412727 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1133/2263] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:54:12 2022] DELETE /identity/v3/users/a6b29b9086f64ea7a7990572567642f7 => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:54:12.943794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5190ce6-d134-4125-a557-7e4dc9a79ea6 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:12.946344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b5190ce6-d134-4125-a557-7e4dc9a79ea6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:12.962819 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5190ce6-d134-4125-a557-7e4dc9a79ea6 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:13.092696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9eda56b-faea-413e-9f91-4ad0c85ced5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:13.093450 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eda56b-faea-413e-9f91-4ad0c85ced5e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:13.093863 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eda56b-faea-413e-9f91-4ad0c85ced5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:13.094355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eda56b-faea-413e-9f91-4ad0c85ced5e admin admin] PATH_INFO: `/v3/projects/ac97d1487c024a68b8ad933d882bb456` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:13.096492 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9eda56b-faea-413e-9f91-4ad0c85ced5e admin admin] RBAC: Authorizing `identity:delete_project(project_id=ac97d1487c024a68b8ad933d882bb456)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:13.100320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9eda56b-faea-413e-9f91-4ad0c85ced5e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:13.296567 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b9eda56b-faea-413e-9f91-4ad0c85ced5e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ac97d1487c024a68b8ad933d882bb456', 'request_id': 'req-b9eda56b-faea-413e-9f91-4ad0c85ced5e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:54:13.298041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=91,keystone:INSERT=5,keystone:DELETE=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:54:13.321139 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b9eda56b-faea-413e-9f91-4ad0c85ced5e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ac97d1487c024a68b8ad933d882bb456', 'request_id': 'req-b9eda56b-faea-413e-9f91-4ad0c85ced5e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:54:13.478224 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b9eda56b-faea-413e-9f91-4ad0c85ced5e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ac97d1487c024a68b8ad933d882bb456', 'request_id': 'req-b9eda56b-faea-413e-9f91-4ad0c85ced5e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:54:13.503209 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b9eda56b-faea-413e-9f91-4ad0c85ced5e admin admin] The token cache is being invalidate because project ac97d1487c024a68b8ad933d882bb456 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:54:13.516278 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b9eda56b-faea-413e-9f91-4ad0c85ced5e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:54:13.516278 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b9eda56b-faea-413e-9f91-4ad0c85ced5e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:54:13.516939 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd43b0b2-7568-4b18-9b32-156710c55778 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:13.516939 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd43b0b2-7568-4b18-9b32-156710c55778 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:13.516939 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd43b0b2-7568-4b18-9b32-156710c55778 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:13.518735 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1131/2264] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:54:12 2022] DELETE /identity/v3/projects/ac97d1487c024a68b8ad933d882bb456 => generated 0 bytes in 566 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:54:13.545547 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-cd43b0b2-7568-4b18-9b32-156710c55778 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:54:13.548331 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-cd43b0b2-7568-4b18-9b32-156710c55778 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:13.671722 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1134/2265] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:54:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:54:13.680611 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85fd0b6-fddb-4d3f-b508-2bc33e602cbf None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:13.680611 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85fd0b6-fddb-4d3f-b508-2bc33e602cbf None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:13.681082 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85fd0b6-fddb-4d3f-b508-2bc33e602cbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:13.703837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-b85fd0b6-fddb-4d3f-b508-2bc33e602cbf None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:54:13.705634 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b85fd0b6-fddb-4d3f-b508-2bc33e602cbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:13.720207 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1132/2266] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:54:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:54:13.721337 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd43b0b2-7568-4b18-9b32-156710c55778 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:13.734318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd43b0b2-7568-4b18-9b32-156710c55778 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:13.741135 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88274b11-e055-40c4-adb6-5aef84a9b05b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:13.741819 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88274b11-e055-40c4-adb6-5aef84a9b05b admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:13.742105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88274b11-e055-40c4-adb6-5aef84a9b05b admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:13.742384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88274b11-e055-40c4-adb6-5aef84a9b05b admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:13.743124 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88274b11-e055-40c4-adb6-5aef84a9b05b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:54:13.743433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88274b11-e055-40c4-adb6-5aef84a9b05b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:13.752492 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88274b11-e055-40c4-adb6-5aef84a9b05b admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:13.777732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1135/2267] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:54:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:54:13.780367 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85fd0b6-fddb-4d3f-b508-2bc33e602cbf None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:13.795566 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85fd0b6-fddb-4d3f-b508-2bc33e602cbf None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:13.816027 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e06af63-b766-46a6-9454-315ee6a4aa6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:13.816971 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e06af63-b766-46a6-9454-315ee6a4aa6c admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:13.817565 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e06af63-b766-46a6-9454-315ee6a4aa6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:13.817855 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e06af63-b766-46a6-9454-315ee6a4aa6c admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:13.818522 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e06af63-b766-46a6-9454-315ee6a4aa6c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:13.820497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e06af63-b766-46a6-9454-315ee6a4aa6c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:13.986989 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1133/2268] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:54:13 2022] POST /identity/v3/projects => generated 371 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:54:13.989421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88274b11-e055-40c4-adb6-5aef84a9b05b admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:14.003106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88274b11-e055-40c4-adb6-5aef84a9b05b admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:14.110356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d6c891-abc5-4b4c-a244-91a9ab2fc5bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:14.111216 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d6c891-abc5-4b4c-a244-91a9ab2fc5bd admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:14.111572 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d6c891-abc5-4b4c-a244-91a9ab2fc5bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:14.111920 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d6c891-abc5-4b4c-a244-91a9ab2fc5bd admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:14.112668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d6c891-abc5-4b4c-a244-91a9ab2fc5bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:14.114774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d6c891-abc5-4b4c-a244-91a9ab2fc5bd admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:14.297514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1136/2269] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:54:13 2022] POST /identity/v3/users => generated 286 bytes in 310 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:54:14.302262 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e06af63-b766-46a6-9454-315ee6a4aa6c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:14.322843 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e06af63-b766-46a6-9454-315ee6a4aa6c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:14.328706 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990655ad-f615-4318-91be-607e901b616b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:14.328846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990655ad-f615-4318-91be-607e901b616b admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:14.329107 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990655ad-f615-4318-91be-607e901b616b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:14.329803 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990655ad-f615-4318-91be-607e901b616b admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:14.330436 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990655ad-f615-4318-91be-607e901b616b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:14.332370 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990655ad-f615-4318-91be-607e901b616b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:14.343565 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1134/2270] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:54:14 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:54:14.348179 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d6c891-abc5-4b4c-a244-91a9ab2fc5bd admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:14.358676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d6c891-abc5-4b4c-a244-91a9ab2fc5bd admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:14.365535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3f8344-7ce7-41ea-8063-0ca3ea705028 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:14.365535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3f8344-7ce7-41ea-8063-0ca3ea705028 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:14.365837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3f8344-7ce7-41ea-8063-0ca3ea705028 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:14.366114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3f8344-7ce7-41ea-8063-0ca3ea705028 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:14.366778 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3f8344-7ce7-41ea-8063-0ca3ea705028 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:14.368602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3f8344-7ce7-41ea-8063-0ca3ea705028 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:14.388794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990655ad-f615-4318-91be-607e901b616b admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:14.389810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1137/2271] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:54:14 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:54:14.402194 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990655ad-f615-4318-91be-607e901b616b admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:14.409312 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d41a6f-0d5d-49e0-bfa7-61efdeeee77b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:14.409312 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d41a6f-0d5d-49e0-bfa7-61efdeeee77b admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:14.409312 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d41a6f-0d5d-49e0-bfa7-61efdeeee77b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:14.409312 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d41a6f-0d5d-49e0-bfa7-61efdeeee77b admin admin] PATH_INFO: `/v3/projects/711d994130664df8849d9336f9542946/users/4ad5c8264bdb44eaa9eeaad8ff193294/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:14.430707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d41a6f-0d5d-49e0-bfa7-61efdeeee77b admin admin] RBAC: Authorizing `identity:create_grant(project_id=711d994130664df8849d9336f9542946, user_id=4ad5c8264bdb44eaa9eeaad8ff193294, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:14.432852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d41a6f-0d5d-49e0-bfa7-61efdeeee77b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:14.594613 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1135/2272] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:54:14 2022] PUT /identity/v3/projects/711d994130664df8849d9336f9542946/users/4ad5c8264bdb44eaa9eeaad8ff193294/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:54:14.607233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7d6696-464a-4fbd-84f1-6e47b9fa7a31 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:14.608293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7d6696-464a-4fbd-84f1-6e47b9fa7a31 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:14.608875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7d6696-464a-4fbd-84f1-6e47b9fa7a31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:14.645886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-8e7d6696-464a-4fbd-84f1-6e47b9fa7a31 None None] MFA Rules not processed for user `4ad5c8264bdb44eaa9eeaad8ff193294`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:54:14.647364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-8e7d6696-464a-4fbd-84f1-6e47b9fa7a31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:14.694473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:54:14.720168 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1138/2273] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:54:14 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:54:14.725226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d41a6f-0d5d-49e0-bfa7-61efdeeee77b admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:14.804553 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d41a6f-0d5d-49e0-bfa7-61efdeeee77b admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:14.810557 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6058b128-495f-4feb-8d8c-852c617335a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:14.811345 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6058b128-495f-4feb-8d8c-852c617335a9 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:14.812038 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6058b128-495f-4feb-8d8c-852c617335a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:14.812038 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6058b128-495f-4feb-8d8c-852c617335a9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:14.818958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6058b128-495f-4feb-8d8c-852c617335a9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:14.821872 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6058b128-495f-4feb-8d8c-852c617335a9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:14.953780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1136/2274] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:54:14 2022] POST /identity/v3/projects => generated 373 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:54:14.967043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7d6696-464a-4fbd-84f1-6e47b9fa7a31 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:14.992595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7d6696-464a-4fbd-84f1-6e47b9fa7a31 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:15.123469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4704c16a-fd20-4b8e-bcd9-5fbf7c4c07e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:15.125712 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4704c16a-fd20-4b8e-bcd9-5fbf7c4c07e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:15.127461 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4704c16a-fd20-4b8e-bcd9-5fbf7c4c07e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:15.127883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4704c16a-fd20-4b8e-bcd9-5fbf7c4c07e5 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:15.131570 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4704c16a-fd20-4b8e-bcd9-5fbf7c4c07e5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:15.136810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4704c16a-fd20-4b8e-bcd9-5fbf7c4c07e5 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:15.318955 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1139/2275] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:54:14 2022] POST /identity/v3/users => generated 293 bytes in 353 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:54:15.323775 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6058b128-495f-4feb-8d8c-852c617335a9 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:15.344080 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614f79cd-e79c-4b13-b21a-cf5152c0b62c None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:15.344368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614f79cd-e79c-4b13-b21a-cf5152c0b62c None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:15.344646 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614f79cd-e79c-4b13-b21a-cf5152c0b62c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:15.345767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6058b128-495f-4feb-8d8c-852c617335a9 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:15.351297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205184cb-7c45-4db4-abda-618958f8ffca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:15.351913 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205184cb-7c45-4db4-abda-618958f8ffca admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:15.352358 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205184cb-7c45-4db4-abda-618958f8ffca admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:15.352636 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205184cb-7c45-4db4-abda-618958f8ffca admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:15.353281 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-205184cb-7c45-4db4-abda-618958f8ffca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:15.354875 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-205184cb-7c45-4db4-abda-618958f8ffca admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:15.368365 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1137/2276] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:54:15 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:54:15.372691 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205184cb-7c45-4db4-abda-618958f8ffca admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:15.385988 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-614f79cd-e79c-4b13-b21a-cf5152c0b62c None None] MFA Rules not processed for user `d21a2934705941ceb91e2f33a4024be1`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:54:15.387277 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-614f79cd-e79c-4b13-b21a-cf5152c0b62c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:15.390129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205184cb-7c45-4db4-abda-618958f8ffca admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:15.395193 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60470fe-011f-495a-9c86-10827f902dd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:15.395852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60470fe-011f-495a-9c86-10827f902dd2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:15.396133 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60470fe-011f-495a-9c86-10827f902dd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:15.396411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60470fe-011f-495a-9c86-10827f902dd2 admin admin] PATH_INFO: `/v3/projects/d1d44d78f6d04c8c9717312798fe73ba/users/e4b8a8e62939475c9df8c7332566bcea/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:15.415181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60470fe-011f-495a-9c86-10827f902dd2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d1d44d78f6d04c8c9717312798fe73ba, user_id=e4b8a8e62939475c9df8c7332566bcea, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:15.419135 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60470fe-011f-495a-9c86-10827f902dd2 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:15.502371 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1140/2277] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:54:15 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:54:15.515580 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614f79cd-e79c-4b13-b21a-cf5152c0b62c None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:15.516955 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-614f79cd-e79c-4b13-b21a-cf5152c0b62c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:15.544824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1138/2278] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:54:15 2022] PUT /identity/v3/projects/d1d44d78f6d04c8c9717312798fe73ba/users/e4b8a8e62939475c9df8c7332566bcea/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:54:15.545869 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60470fe-011f-495a-9c86-10827f902dd2 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:15.584973 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:54:15.602785 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614f79cd-e79c-4b13-b21a-cf5152c0b62c None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:15.638369 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60470fe-011f-495a-9c86-10827f902dd2 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:15.646995 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec76c598-0d31-4ca0-8e91-b0ff67b5163d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:15.648754 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec76c598-0d31-4ca0-8e91-b0ff67b5163d admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:15.649040 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec76c598-0d31-4ca0-8e91-b0ff67b5163d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:15.649465 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec76c598-0d31-4ca0-8e91-b0ff67b5163d admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:15.650136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec76c598-0d31-4ca0-8e91-b0ff67b5163d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:15.652041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec76c598-0d31-4ca0-8e91-b0ff67b5163d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:15.668785 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1139/2279] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:54:15 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:54:15.676708 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec76c598-0d31-4ca0-8e91-b0ff67b5163d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:15.686623 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2104a7-0535-47ad-b009-b9106829aafd service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ceb9fc31c345ccabaee8f30a3982a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:15.687355 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2104a7-0535-47ad-b009-b9106829aafd service glance] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:15.687763 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2104a7-0535-47ad-b009-b9106829aafd service glance] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:15.687875 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec76c598-0d31-4ca0-8e91-b0ff67b5163d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:15.688161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2104a7-0535-47ad-b009-b9106829aafd service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:15.690442 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-8f2104a7-0535-47ad-b009-b9106829aafd service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:15.693851 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d426398-fa66-4a29-aa69-ab95a4c6b2f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:15.694575 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d426398-fa66-4a29-aa69-ab95a4c6b2f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:15.694856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d426398-fa66-4a29-aa69-ab95a4c6b2f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:15.695163 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d426398-fa66-4a29-aa69-ab95a4c6b2f7 admin admin] PATH_INFO: `/v3/projects/d1d44d78f6d04c8c9717312798fe73ba/users/e4b8a8e62939475c9df8c7332566bcea/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:15.697155 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d426398-fa66-4a29-aa69-ab95a4c6b2f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d1d44d78f6d04c8c9717312798fe73ba, user_id=e4b8a8e62939475c9df8c7332566bcea, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:15.699934 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d426398-fa66-4a29-aa69-ab95a4c6b2f7 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:15.720791 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1140/2280] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:54:15 2022] PUT /identity/v3/projects/d1d44d78f6d04c8c9717312798fe73ba/users/e4b8a8e62939475c9df8c7332566bcea/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:54:15.723366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2d66e0-5157-434a-a622-4c19a84b83ea None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:15.723662 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2d66e0-5157-434a-a622-4c19a84b83ea None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:15.723941 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2d66e0-5157-434a-a622-4c19a84b83ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:15.768190 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-bd2d66e0-5157-434a-a622-4c19a84b83ea None None] MFA Rules not processed for user `e4b8a8e62939475c9df8c7332566bcea`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:54:15.768190 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-bd2d66e0-5157-434a-a622-4c19a84b83ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:15.768451 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2104a7-0535-47ad-b009-b9106829aafd service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:15.768451 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2104a7-0535-47ad-b009-b9106829aafd service glance] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:15.778019 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1141/2281] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:54:15 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 263 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:54:15.843541 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1141/2282] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:54:15 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:54:15.852901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2104a7-0535-47ad-b009-b9106829aafd service glance] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:15.854384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-8f2104a7-0535-47ad-b009-b9106829aafd service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:15.923490 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2104a7-0535-47ad-b009-b9106829aafd service glance] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:15.994487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e7ff7d-ecff-4fb8-97e5-7f8597c4b5c0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:15.995377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e7ff7d-ecff-4fb8-97e5-7f8597c4b5c0 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:15.996046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e7ff7d-ecff-4fb8-97e5-7f8597c4b5c0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:15.996046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e7ff7d-ecff-4fb8-97e5-7f8597c4b5c0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:16.009435 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e7ff7d-ecff-4fb8-97e5-7f8597c4b5c0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:16.009435 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e7ff7d-ecff-4fb8-97e5-7f8597c4b5c0 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:16.021069 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1142/2283] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:54:15 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:54:16.118521 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d54796e-37f6-48b2-a3d9-38dc685203b8 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:16.118806 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d54796e-37f6-48b2-a3d9-38dc685203b8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:16.119117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d54796e-37f6-48b2-a3d9-38dc685203b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:16.138745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8d54796e-37f6-48b2-a3d9-38dc685203b8 None None] MFA Rules not processed for user `d40810f61ead4c3da414194b325da80a`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:54:16.139982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8d54796e-37f6-48b2-a3d9-38dc685203b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:16.181417 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:54:16.209024 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1142/2284] 10.222.0.49 () {58 vars in 1009 bytes} [Fri Sep 16 03:54:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:54:16.264582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e7ff7d-ecff-4fb8-97e5-7f8597c4b5c0 service cinder] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:16.274285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e7ff7d-ecff-4fb8-97e5-7f8597c4b5c0 service cinder] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:16.278124 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f104654-3085-49c2-b8c7-bc1962687ac8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:16.278714 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f104654-3085-49c2-b8c7-bc1962687ac8 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:16.279019 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f104654-3085-49c2-b8c7-bc1962687ac8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:16.279293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f104654-3085-49c2-b8c7-bc1962687ac8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:16.335689 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f104654-3085-49c2-b8c7-bc1962687ac8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:16.337168 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f104654-3085-49c2-b8c7-bc1962687ac8 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:16.346118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1143/2285] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:54:16 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:54:16.418152 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d54796e-37f6-48b2-a3d9-38dc685203b8 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:16.420138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8d54796e-37f6-48b2-a3d9-38dc685203b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:16.507453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d54796e-37f6-48b2-a3d9-38dc685203b8 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:16.512446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3105892-9c6b-4255-af6b-3c7934bab85b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa30da9972e140d7905c522576681429', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:16.513060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3105892-9c6b-4255-af6b-3c7934bab85b service swift] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:16.513342 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3105892-9c6b-4255-af6b-3c7934bab85b service swift] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:16.513700 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3105892-9c6b-4255-af6b-3c7934bab85b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:16.522778 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3105892-9c6b-4255-af6b-3c7934bab85b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:16.524430 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3105892-9c6b-4255-af6b-3c7934bab85b service swift] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:16.532695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1143/2286] 10.222.0.49 () {60 vars in 1384 bytes} [Fri Sep 16 03:54:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 03:54:17.438841 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:54:17.553351 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:54:21.321136 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37206f21-c0ae-4139-9539-3e2d79fe5b69 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:21.321739 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37206f21-c0ae-4139-9539-3e2d79fe5b69 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:21.321739 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37206f21-c0ae-4139-9539-3e2d79fe5b69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:21.377206 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-37206f21-c0ae-4139-9539-3e2d79fe5b69 None None] MFA Rules not processed for user `ff7df9be8475405eba924b59a7bd6bb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:54:21.379536 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-37206f21-c0ae-4139-9539-3e2d79fe5b69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:21.461724 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1144/2287] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:54:21 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:54:28.934490 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=15 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:54:29.341445 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651a6f0a-0c15-4e86-900a-c440ca5bad06 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:29.341740 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651a6f0a-0c15-4e86-900a-c440ca5bad06 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:29.342022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651a6f0a-0c15-4e86-900a-c440ca5bad06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:29.362314 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-651a6f0a-0c15-4e86-900a-c440ca5bad06 None None] MFA Rules not processed for user `269dbbd6d926496d8c5c704832de5763`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:54:29.364233 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-651a6f0a-0c15-4e86-900a-c440ca5bad06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:29.423306 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1144/2288] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:54:29 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:54:29.685387 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37206f21-c0ae-4139-9539-3e2d79fe5b69 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:29.686845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-37206f21-c0ae-4139-9539-3e2d79fe5b69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:29.748927 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37206f21-c0ae-4139-9539-3e2d79fe5b69 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:29.752834 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fb7f36-6de5-4122-a2ff-e4a9f4e4ee4f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:29.753624 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fb7f36-6de5-4122-a2ff-e4a9f4e4ee4f service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:29.754021 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fb7f36-6de5-4122-a2ff-e4a9f4e4ee4f service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:29.754458 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fb7f36-6de5-4122-a2ff-e4a9f4e4ee4f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:29.756583 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e5fb7f36-6de5-4122-a2ff-e4a9f4e4ee4f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:29.766692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fb7f36-6de5-4122-a2ff-e4a9f4e4ee4f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:29.768159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fb7f36-6de5-4122-a2ff-e4a9f4e4ee4f service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:29.776993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1145/2289] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:54:29 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:54:33.466834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651a6f0a-0c15-4e86-900a-c440ca5bad06 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:33.468347 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-651a6f0a-0c15-4e86-900a-c440ca5bad06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:33.559933 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651a6f0a-0c15-4e86-900a-c440ca5bad06 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:33.565810 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e927e0c5-53d0-4440-ad17-2b7bbf859f68 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:33.566553 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e927e0c5-53d0-4440-ad17-2b7bbf859f68 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:33.566897 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e927e0c5-53d0-4440-ad17-2b7bbf859f68 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:33.567303 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e927e0c5-53d0-4440-ad17-2b7bbf859f68 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:33.569584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e927e0c5-53d0-4440-ad17-2b7bbf859f68 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:33.592536 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e927e0c5-53d0-4440-ad17-2b7bbf859f68 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:33.594221 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e927e0c5-53d0-4440-ad17-2b7bbf859f68 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:33.609467 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1145/2290] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:54:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:54:33.656411 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22bbd68-440c-4a4c-9bca-e51659f987fb None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:33.656916 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22bbd68-440c-4a4c-9bca-e51659f987fb None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:33.659034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22bbd68-440c-4a4c-9bca-e51659f987fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:33.691201 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-b22bbd68-440c-4a4c-9bca-e51659f987fb None None] MFA Rules not processed for user `d40810f61ead4c3da414194b325da80a`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:54:33.692744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b22bbd68-440c-4a4c-9bca-e51659f987fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:33.727045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1146/2291] 10.222.0.49 () {58 vars in 1009 bytes} [Fri Sep 16 03:54:33 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:54:33.733744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e927e0c5-53d0-4440-ad17-2b7bbf859f68 service nova] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:33.752023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e927e0c5-53d0-4440-ad17-2b7bbf859f68 service nova] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:33.764542 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36579075-9e20-4aea-af5e-c3ca2538bb72 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa30da9972e140d7905c522576681429', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:33.764542 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36579075-9e20-4aea-af5e-c3ca2538bb72 service swift] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:33.764542 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36579075-9e20-4aea-af5e-c3ca2538bb72 service swift] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:33.764542 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36579075-9e20-4aea-af5e-c3ca2538bb72 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:33.792175 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36579075-9e20-4aea-af5e-c3ca2538bb72 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:33.793778 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36579075-9e20-4aea-af5e-c3ca2538bb72 service swift] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:33.803271 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1146/2292] 10.222.0.49 () {60 vars in 1384 bytes} [Fri Sep 16 03:54:33 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 03:54:43.684765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=70 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:54:43.799271 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=65 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:54:50.262692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504ab89a-7ce6-45a1-8af9-109a1f24d6cc None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:50.263213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504ab89a-7ce6-45a1-8af9-109a1f24d6cc None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:50.263338 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504ab89a-7ce6-45a1-8af9-109a1f24d6cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:50.303472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-504ab89a-7ce6-45a1-8af9-109a1f24d6cc None None] MFA Rules not processed for user `d40810f61ead4c3da414194b325da80a`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:54:50.304687 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-504ab89a-7ce6-45a1-8af9-109a1f24d6cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:54:50.331258 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1147/2293] 10.222.0.49 () {58 vars in 1009 bytes} [Fri Sep 16 03:54:50 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:54:50.344307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36579075-9e20-4aea-af5e-c3ca2538bb72 service swift] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:54:50.384137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36579075-9e20-4aea-af5e-c3ca2538bb72 service swift] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:54:50.389061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14816a57-f3f5-4672-a31f-484ce475e106 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa30da9972e140d7905c522576681429', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:54:50.389660 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14816a57-f3f5-4672-a31f-484ce475e106 service swift] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:54:50.389923 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14816a57-f3f5-4672-a31f-484ce475e106 service swift] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:54:50.390191 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14816a57-f3f5-4672-a31f-484ce475e106 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:54:50.398528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14816a57-f3f5-4672-a31f-484ce475e106 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:54:50.400172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14816a57-f3f5-4672-a31f-484ce475e106 service swift] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:54:50.407972 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1147/2294] 10.222.0.49 () {60 vars in 1384 bytes} [Fri Sep 16 03:54:50 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 03:55:00.329392 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=20 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:00.404936 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=20 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:10.138915 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504ab89a-7ce6-45a1-8af9-109a1f24d6cc None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:10.158344 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504ab89a-7ce6-45a1-8af9-109a1f24d6cc None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:10.162512 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd9ae52-88af-453e-873d-62eae2516d02 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:10.163155 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd9ae52-88af-453e-873d-62eae2516d02 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:10.163433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd9ae52-88af-453e-873d-62eae2516d02 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:10.163703 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd9ae52-88af-453e-873d-62eae2516d02 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:10.225117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd9ae52-88af-453e-873d-62eae2516d02 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:10.226539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd9ae52-88af-453e-873d-62eae2516d02 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:10.241076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1148/2295] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:10 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:55:12.832044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14816a57-f3f5-4672-a31f-484ce475e106 service swift] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:12.968288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14816a57-f3f5-4672-a31f-484ce475e106 service swift] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:12.980102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555802a2-3231-4e48-87df-341079b207bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:12.986962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555802a2-3231-4e48-87df-341079b207bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:12.987107 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555802a2-3231-4e48-87df-341079b207bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:12.987409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555802a2-3231-4e48-87df-341079b207bb admin admin] PATH_INFO: `/v3/users/4ad5c8264bdb44eaa9eeaad8ff193294` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:12.990383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555802a2-3231-4e48-87df-341079b207bb admin admin] RBAC: Authorizing `identity:delete_user(user_id=4ad5c8264bdb44eaa9eeaad8ff193294)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:12.996818 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555802a2-3231-4e48-87df-341079b207bb admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:13.252529 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-555802a2-3231-4e48-87df-341079b207bb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4ad5c8264bdb44eaa9eeaad8ff193294'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:13.266751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-555802a2-3231-4e48-87df-341079b207bb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4ad5c8264bdb44eaa9eeaad8ff193294'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:13.380726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-555802a2-3231-4e48-87df-341079b207bb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4ad5c8264bdb44eaa9eeaad8ff193294'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:13.381581 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-555802a2-3231-4e48-87df-341079b207bb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4ad5c8264bdb44eaa9eeaad8ff193294'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:13.399491 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-555802a2-3231-4e48-87df-341079b207bb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4ad5c8264bdb44eaa9eeaad8ff193294'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:13.409307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1148/2296] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:55:12 2022] DELETE /identity/v3/users/4ad5c8264bdb44eaa9eeaad8ff193294 => generated 0 bytes in 578 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:55:13.413554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce41c51-7237-40fe-be98-12763950c732 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:13.414092 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce41c51-7237-40fe-be98-12763950c732 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:13.414092 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce41c51-7237-40fe-be98-12763950c732 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:13.456834 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-1ce41c51-7237-40fe-be98-12763950c732 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:13.458343 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1ce41c51-7237-40fe-be98-12763950c732 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:13.518274 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1149/2297] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:55:13.528539 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555802a2-3231-4e48-87df-341079b207bb admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:13.531458 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-555802a2-3231-4e48-87df-341079b207bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:13.629180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555802a2-3231-4e48-87df-341079b207bb admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:13.633821 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df920dbf-7b12-4e7d-9785-05c44288cd23 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:13.634449 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df920dbf-7b12-4e7d-9785-05c44288cd23 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:13.634724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df920dbf-7b12-4e7d-9785-05c44288cd23 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:13.635076 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df920dbf-7b12-4e7d-9785-05c44288cd23 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:13.641118 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:13.644766 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df920dbf-7b12-4e7d-9785-05c44288cd23 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:13.646334 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df920dbf-7b12-4e7d-9785-05c44288cd23 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:13.655727 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1149/2298] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:55:14.130530 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce41c51-7237-40fe-be98-12763950c732 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:14.131990 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1ce41c51-7237-40fe-be98-12763950c732 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:14.154705 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce41c51-7237-40fe-be98-12763950c732 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:14.164307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd38721-8825-4d6a-9baf-7a04e04ee229 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:14.164307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd38721-8825-4d6a-9baf-7a04e04ee229 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:14.164307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd38721-8825-4d6a-9baf-7a04e04ee229 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:14.164307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd38721-8825-4d6a-9baf-7a04e04ee229 admin admin] PATH_INFO: `/v3/projects/711d994130664df8849d9336f9542946` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:14.166330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd38721-8825-4d6a-9baf-7a04e04ee229 admin admin] RBAC: Authorizing `identity:delete_project(project_id=711d994130664df8849d9336f9542946)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:14.169533 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd38721-8825-4d6a-9baf-7a04e04ee229 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:14.330959 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6cd38721-8825-4d6a-9baf-7a04e04ee229 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '711d994130664df8849d9336f9542946', 'request_id': 'req-6cd38721-8825-4d6a-9baf-7a04e04ee229'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:14.410458 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6cd38721-8825-4d6a-9baf-7a04e04ee229 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '711d994130664df8849d9336f9542946', 'request_id': 'req-6cd38721-8825-4d6a-9baf-7a04e04ee229'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:14.417148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:14.418568 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6cd38721-8825-4d6a-9baf-7a04e04ee229 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '711d994130664df8849d9336f9542946', 'request_id': 'req-6cd38721-8825-4d6a-9baf-7a04e04ee229'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:14.430523 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6cd38721-8825-4d6a-9baf-7a04e04ee229 admin admin] The token cache is being invalidate because project 711d994130664df8849d9336f9542946 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:55:14.430902 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6cd38721-8825-4d6a-9baf-7a04e04ee229 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:14.431670 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6cd38721-8825-4d6a-9baf-7a04e04ee229 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:14.433577 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1150/2299] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:55:14 2022] DELETE /identity/v3/projects/711d994130664df8849d9336f9542946 => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:14.437627 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df920dbf-7b12-4e7d-9785-05c44288cd23 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:14.448031 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df920dbf-7b12-4e7d-9785-05c44288cd23 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:14.514392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b689401d-64af-4663-8a9e-3a2f148b34dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:14.515091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b689401d-64af-4663-8a9e-3a2f148b34dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:14.515612 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b689401d-64af-4663-8a9e-3a2f148b34dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:14.515899 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b689401d-64af-4663-8a9e-3a2f148b34dc admin admin] PATH_INFO: `/v3/users/e4b8a8e62939475c9df8c7332566bcea` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:14.517205 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b689401d-64af-4663-8a9e-3a2f148b34dc admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4b8a8e62939475c9df8c7332566bcea)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:14.519225 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b689401d-64af-4663-8a9e-3a2f148b34dc admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:14.753863 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b689401d-64af-4663-8a9e-3a2f148b34dc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4b8a8e62939475c9df8c7332566bcea'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:14.766300 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b689401d-64af-4663-8a9e-3a2f148b34dc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4b8a8e62939475c9df8c7332566bcea'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:14.860617 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b689401d-64af-4663-8a9e-3a2f148b34dc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4b8a8e62939475c9df8c7332566bcea'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:14.861235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b689401d-64af-4663-8a9e-3a2f148b34dc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4b8a8e62939475c9df8c7332566bcea'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:14.869297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b689401d-64af-4663-8a9e-3a2f148b34dc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4b8a8e62939475c9df8c7332566bcea'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:14.887457 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1150/2300] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:55:14 2022] DELETE /identity/v3/users/e4b8a8e62939475c9df8c7332566bcea => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:14.960326 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d093e6-0abc-4874-bb87-15920123bcef None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:14.960608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d093e6-0abc-4874-bb87-15920123bcef None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:14.961056 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d093e6-0abc-4874-bb87-15920123bcef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:14.980030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-b8d093e6-0abc-4874-bb87-15920123bcef None None] MFA Rules not processed for user `d40810f61ead4c3da414194b325da80a`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:14.981245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b8d093e6-0abc-4874-bb87-15920123bcef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:15.034421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1151/2301] 10.222.0.49 () {58 vars in 1009 bytes} [Fri Sep 16 03:55:14 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:55:15.046710 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b689401d-64af-4663-8a9e-3a2f148b34dc admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:15.048154 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b689401d-64af-4663-8a9e-3a2f148b34dc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:15.121572 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b689401d-64af-4663-8a9e-3a2f148b34dc admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:15.126888 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba71381-19ed-4cff-8a18-70e09f429df6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa30da9972e140d7905c522576681429', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:15.127867 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba71381-19ed-4cff-8a18-70e09f429df6 service swift] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:15.128150 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba71381-19ed-4cff-8a18-70e09f429df6 service swift] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:15.128425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba71381-19ed-4cff-8a18-70e09f429df6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:15.141059 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba71381-19ed-4cff-8a18-70e09f429df6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:15.141059 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba71381-19ed-4cff-8a18-70e09f429df6 service swift] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:15.150217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1151/2302] 10.222.0.49 () {60 vars in 1384 bytes} [Fri Sep 16 03:55:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 03:55:15.800188 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d093e6-0abc-4874-bb87-15920123bcef None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:15.801646 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b8d093e6-0abc-4874-bb87-15920123bcef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:15.900588 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d093e6-0abc-4874-bb87-15920123bcef None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:15.905124 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61325e21-64ec-4e84-9707-9fce9845b5ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:15.905763 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61325e21-64ec-4e84-9707-9fce9845b5ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:15.906044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61325e21-64ec-4e84-9707-9fce9845b5ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:15.906318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61325e21-64ec-4e84-9707-9fce9845b5ce admin admin] PATH_INFO: `/v3/projects/d1d44d78f6d04c8c9717312798fe73ba` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:15.907644 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61325e21-64ec-4e84-9707-9fce9845b5ce admin admin] RBAC: Authorizing `identity:delete_project(project_id=d1d44d78f6d04c8c9717312798fe73ba)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:15.909633 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61325e21-64ec-4e84-9707-9fce9845b5ce admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:16.045962 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-61325e21-64ec-4e84-9707-9fce9845b5ce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd1d44d78f6d04c8c9717312798fe73ba', 'request_id': 'req-61325e21-64ec-4e84-9707-9fce9845b5ce'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:16.128309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-61325e21-64ec-4e84-9707-9fce9845b5ce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd1d44d78f6d04c8c9717312798fe73ba', 'request_id': 'req-61325e21-64ec-4e84-9707-9fce9845b5ce'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:16.135427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-61325e21-64ec-4e84-9707-9fce9845b5ce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd1d44d78f6d04c8c9717312798fe73ba', 'request_id': 'req-61325e21-64ec-4e84-9707-9fce9845b5ce'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:16.149993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-61325e21-64ec-4e84-9707-9fce9845b5ce admin admin] The token cache is being invalidate because project d1d44d78f6d04c8c9717312798fe73ba was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:55:16.150314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-61325e21-64ec-4e84-9707-9fce9845b5ce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:16.151136 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-61325e21-64ec-4e84-9707-9fce9845b5ce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:16.153083 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1152/2303] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:55:15 2022] DELETE /identity/v3/projects/d1d44d78f6d04c8c9717312798fe73ba => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:16.158129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e820569-136a-49f0-b0f6-953239df14d9 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:16.158639 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e820569-136a-49f0-b0f6-953239df14d9 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:16.158741 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e820569-136a-49f0-b0f6-953239df14d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:16.170333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:16.243512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8e820569-136a-49f0-b0f6-953239df14d9 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:16.244726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8e820569-136a-49f0-b0f6-953239df14d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:16.356073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1152/2304] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:55:16.364963 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc49a5e-cfeb-44a2-8724-25e5ce44cc07 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:16.365343 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc49a5e-cfeb-44a2-8724-25e5ce44cc07 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:16.365687 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc49a5e-cfeb-44a2-8724-25e5ce44cc07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:16.389823 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-bfc49a5e-cfeb-44a2-8724-25e5ce44cc07 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:16.391109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-bfc49a5e-cfeb-44a2-8724-25e5ce44cc07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:16.408863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1153/2305] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:55:16.414360 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e820569-136a-49f0-b0f6-953239df14d9 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:16.424405 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e820569-136a-49f0-b0f6-953239df14d9 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:16.432682 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bbd651-2cde-4bb2-a638-78559b42145e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:16.433336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bbd651-2cde-4bb2-a638-78559b42145e admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:16.433622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bbd651-2cde-4bb2-a638-78559b42145e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:16.434234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bbd651-2cde-4bb2-a638-78559b42145e admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:16.434665 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bbd651-2cde-4bb2-a638-78559b42145e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:55:16.435071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bbd651-2cde-4bb2-a638-78559b42145e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:16.436970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bbd651-2cde-4bb2-a638-78559b42145e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:16.454554 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1153/2306] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:55:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:55:16.455074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc49a5e-cfeb-44a2-8724-25e5ce44cc07 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:16.467729 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc49a5e-cfeb-44a2-8724-25e5ce44cc07 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:16.474353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac61a82-099a-46f4-b557-f0e3830002e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:16.475023 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac61a82-099a-46f4-b557-f0e3830002e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:16.475311 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac61a82-099a-46f4-b557-f0e3830002e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:16.475585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac61a82-099a-46f4-b557-f0e3830002e7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:16.476872 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac61a82-099a-46f4-b557-f0e3830002e7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:16.479563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac61a82-099a-46f4-b557-f0e3830002e7 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:16.504963 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:16.594666 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1154/2307] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:55:16 2022] POST /identity/v3/projects => generated 363 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:55:16.598898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bbd651-2cde-4bb2-a638-78559b42145e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:16.612343 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bbd651-2cde-4bb2-a638-78559b42145e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:16.681421 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2546d52b-65b5-47b4-9936-c81a64903389 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:16.682211 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2546d52b-65b5-47b4-9936-c81a64903389 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:16.683061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2546d52b-65b5-47b4-9936-c81a64903389 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:16.683409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2546d52b-65b5-47b4-9936-c81a64903389 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:16.684115 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2546d52b-65b5-47b4-9936-c81a64903389 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:16.686187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2546d52b-65b5-47b4-9936-c81a64903389 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:16.782774 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1154/2308] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:55:16 2022] POST /identity/v3/users => generated 282 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:55:16.787845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac61a82-099a-46f4-b557-f0e3830002e7 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:16.818001 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac61a82-099a-46f4-b557-f0e3830002e7 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:16.835446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c163dc-8fe9-4a37-9c03-a42f911e354e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:16.836347 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c163dc-8fe9-4a37-9c03-a42f911e354e admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:16.837372 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c163dc-8fe9-4a37-9c03-a42f911e354e admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:16.837863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c163dc-8fe9-4a37-9c03-a42f911e354e admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:16.838732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c163dc-8fe9-4a37-9c03-a42f911e354e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:16.844730 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c163dc-8fe9-4a37-9c03-a42f911e354e admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:16.881558 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1155/2309] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:55:16 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:55:16.885503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2546d52b-65b5-47b4-9936-c81a64903389 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:16.897143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2546d52b-65b5-47b4-9936-c81a64903389 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:16.913209 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8e954f-5616-4349-8805-0714af67d2b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:16.913963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8e954f-5616-4349-8805-0714af67d2b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:16.914497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8e954f-5616-4349-8805-0714af67d2b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:16.914885 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8e954f-5616-4349-8805-0714af67d2b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:16.915670 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8e954f-5616-4349-8805-0714af67d2b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:16.919756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8e954f-5616-4349-8805-0714af67d2b7 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:16.951396 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1155/2310] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:55:16 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:55:16.955638 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c163dc-8fe9-4a37-9c03-a42f911e354e admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:16.981032 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c163dc-8fe9-4a37-9c03-a42f911e354e admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:16.992763 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01312f7-3015-45aa-a837-ab5b2c2aa7d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:17.000924 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01312f7-3015-45aa-a837-ab5b2c2aa7d8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:17.000924 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01312f7-3015-45aa-a837-ab5b2c2aa7d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:17.000924 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01312f7-3015-45aa-a837-ab5b2c2aa7d8 admin admin] PATH_INFO: `/v3/projects/78c2e054024346ecab252245a8ece557/users/d297f4947ce64cad93940e04c6a13a00/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:17.028047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01312f7-3015-45aa-a837-ab5b2c2aa7d8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=78c2e054024346ecab252245a8ece557, user_id=d297f4947ce64cad93940e04c6a13a00, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:17.031456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01312f7-3015-45aa-a837-ab5b2c2aa7d8 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:17.120742 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1156/2311] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:55:16 2022] PUT /identity/v3/projects/78c2e054024346ecab252245a8ece557/users/d297f4947ce64cad93940e04c6a13a00/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:17.131995 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4bd5bb-c4e3-4d08-9aa1-4e9fb6b347bb None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:17.132602 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4bd5bb-c4e3-4d08-9aa1-4e9fb6b347bb None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:17.133247 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4bd5bb-c4e3-4d08-9aa1-4e9fb6b347bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:17.210647 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ea4bd5bb-c4e3-4d08-9aa1-4e9fb6b347bb None None] MFA Rules not processed for user `d297f4947ce64cad93940e04c6a13a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:17.211939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ea4bd5bb-c4e3-4d08-9aa1-4e9fb6b347bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:17.291396 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:17.403306 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1156/2312] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:17 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 268 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:55:17.404733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01312f7-3015-45aa-a837-ab5b2c2aa7d8 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:17.501428 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01312f7-3015-45aa-a837-ab5b2c2aa7d8 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:17.508591 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce68fba7-47b4-4e0c-8063-5aa1fc9ba0ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:17.509398 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce68fba7-47b4-4e0c-8063-5aa1fc9ba0ae admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:17.509690 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce68fba7-47b4-4e0c-8063-5aa1fc9ba0ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:17.510487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce68fba7-47b4-4e0c-8063-5aa1fc9ba0ae admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:17.511526 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce68fba7-47b4-4e0c-8063-5aa1fc9ba0ae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:17.514465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce68fba7-47b4-4e0c-8063-5aa1fc9ba0ae admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:17.682430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1157/2313] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:55:17 2022] POST /identity/v3/projects => generated 365 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:55:17.685759 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4bd5bb-c4e3-4d08-9aa1-4e9fb6b347bb None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:17.697938 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4bd5bb-c4e3-4d08-9aa1-4e9fb6b347bb None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:17.786834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682592cc-2eba-4a02-b622-7f67cf76f17c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:17.792228 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682592cc-2eba-4a02-b622-7f67cf76f17c admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:17.792228 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682592cc-2eba-4a02-b622-7f67cf76f17c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:17.792228 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682592cc-2eba-4a02-b622-7f67cf76f17c admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:17.792228 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682592cc-2eba-4a02-b622-7f67cf76f17c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:17.792228 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682592cc-2eba-4a02-b622-7f67cf76f17c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:18.240709 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1157/2314] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:55:17 2022] POST /identity/v3/users => generated 289 bytes in 555 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:55:18.244786 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce68fba7-47b4-4e0c-8063-5aa1fc9ba0ae admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:18.256603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce68fba7-47b4-4e0c-8063-5aa1fc9ba0ae admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:18.262105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb5c4bb-69f5-40f4-a741-22c6a23e9dd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:18.262720 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb5c4bb-69f5-40f4-a741-22c6a23e9dd2 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:18.263039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb5c4bb-69f5-40f4-a741-22c6a23e9dd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:18.263311 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb5c4bb-69f5-40f4-a741-22c6a23e9dd2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:18.264315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb5c4bb-69f5-40f4-a741-22c6a23e9dd2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:18.266371 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb5c4bb-69f5-40f4-a741-22c6a23e9dd2 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:18.283184 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1158/2315] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:55:18 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:55:18.288123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682592cc-2eba-4a02-b622-7f67cf76f17c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:18.299183 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682592cc-2eba-4a02-b622-7f67cf76f17c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:18.305530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49063346-195a-4e86-a111-120438c0cf64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:18.309991 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49063346-195a-4e86-a111-120438c0cf64 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:18.309991 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49063346-195a-4e86-a111-120438c0cf64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:18.309991 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49063346-195a-4e86-a111-120438c0cf64 admin admin] PATH_INFO: `/v3/projects/14406d9a93fe450aa372e291225a5612/users/6b2893eb3df149409e051fba1b5388f3/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:18.330901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49063346-195a-4e86-a111-120438c0cf64 admin admin] RBAC: Authorizing `identity:create_grant(project_id=14406d9a93fe450aa372e291225a5612, user_id=6b2893eb3df149409e051fba1b5388f3, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:18.334783 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49063346-195a-4e86-a111-120438c0cf64 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:18.439623 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1158/2316] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:55:18 2022] PUT /identity/v3/projects/14406d9a93fe450aa372e291225a5612/users/6b2893eb3df149409e051fba1b5388f3/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:18.443867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb5c4bb-69f5-40f4-a741-22c6a23e9dd2 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:18.465269 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb5c4bb-69f5-40f4-a741-22c6a23e9dd2 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:18.568282 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769bde23-7551-44c2-8281-9620ee8d24f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:18.568561 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769bde23-7551-44c2-8281-9620ee8d24f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:18.569008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769bde23-7551-44c2-8281-9620ee8d24f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:18.569389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769bde23-7551-44c2-8281-9620ee8d24f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:18.570135 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769bde23-7551-44c2-8281-9620ee8d24f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:18.572718 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769bde23-7551-44c2-8281-9620ee8d24f4 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:18.584543 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1159/2317] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:55:18 2022] GET /identity/v3/roles => generated 1420 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:55:18.588835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49063346-195a-4e86-a111-120438c0cf64 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:18.600451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49063346-195a-4e86-a111-120438c0cf64 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:18.605385 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e5c610-df87-48d2-a97e-6e0da8b06c68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:18.606152 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e5c610-df87-48d2-a97e-6e0da8b06c68 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:18.606422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e5c610-df87-48d2-a97e-6e0da8b06c68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:18.606728 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e5c610-df87-48d2-a97e-6e0da8b06c68 admin admin] PATH_INFO: `/v3/projects/14406d9a93fe450aa372e291225a5612/users/6b2893eb3df149409e051fba1b5388f3/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:18.608749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e5c610-df87-48d2-a97e-6e0da8b06c68 admin admin] RBAC: Authorizing `identity:create_grant(project_id=14406d9a93fe450aa372e291225a5612, user_id=6b2893eb3df149409e051fba1b5388f3, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:18.611624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e5c610-df87-48d2-a97e-6e0da8b06c68 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:18.624645 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1159/2318] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:55:18 2022] PUT /identity/v3/projects/14406d9a93fe450aa372e291225a5612/users/6b2893eb3df149409e051fba1b5388f3/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:18.638139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05ec8b7-818a-4dc0-88c7-7bdc12cc0cfc None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:18.638556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05ec8b7-818a-4dc0-88c7-7bdc12cc0cfc None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:18.638902 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05ec8b7-818a-4dc0-88c7-7bdc12cc0cfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:18.671280 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-b05ec8b7-818a-4dc0-88c7-7bdc12cc0cfc None None] MFA Rules not processed for user `6b2893eb3df149409e051fba1b5388f3`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:18.672602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b05ec8b7-818a-4dc0-88c7-7bdc12cc0cfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:18.688984 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e5c610-df87-48d2-a97e-6e0da8b06c68 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:18.690859 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-38e5c610-df87-48d2-a97e-6e0da8b06c68 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:18.692451 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:18.783115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1160/2319] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:18 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:55:18.790634 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05ec8b7-818a-4dc0-88c7-7bdc12cc0cfc None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:18.793212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b05ec8b7-818a-4dc0-88c7-7bdc12cc0cfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:18.830179 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e5c610-df87-48d2-a97e-6e0da8b06c68 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:18.835780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac45743-0b32-402d-abf6-0a879b6e6a3b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:18.836765 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac45743-0b32-402d-abf6-0a879b6e6a3b service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:18.837343 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac45743-0b32-402d-abf6-0a879b6e6a3b service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:18.837795 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac45743-0b32-402d-abf6-0a879b6e6a3b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:18.840138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cac45743-0b32-402d-abf6-0a879b6e6a3b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:18.897486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05ec8b7-818a-4dc0-88c7-7bdc12cc0cfc None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:18.899725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:18.903109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8073c7ae-8dde-48f5-8c50-59eb69b5374a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:18.904049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8073c7ae-8dde-48f5-8c50-59eb69b5374a service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:18.904501 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8073c7ae-8dde-48f5-8c50-59eb69b5374a service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:18.904997 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8073c7ae-8dde-48f5-8c50-59eb69b5374a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:18.916912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8073c7ae-8dde-48f5-8c50-59eb69b5374a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:18.919802 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8073c7ae-8dde-48f5-8c50-59eb69b5374a service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:18.924258 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac45743-0b32-402d-abf6-0a879b6e6a3b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:18.926116 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac45743-0b32-402d-abf6-0a879b6e6a3b service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:18.933066 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1161/2320] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:18 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:55:18.939321 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1160/2321] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:18 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 250 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:55:19.383504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8073c7ae-8dde-48f5-8c50-59eb69b5374a service cinder] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:19.411046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8073c7ae-8dde-48f5-8c50-59eb69b5374a service cinder] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:19.422838 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8feb3e26-7737-4f83-bc59-4ffaa7a89262 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:19.423642 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8feb3e26-7737-4f83-bc59-4ffaa7a89262 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:19.424013 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8feb3e26-7737-4f83-bc59-4ffaa7a89262 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:19.424382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8feb3e26-7737-4f83-bc59-4ffaa7a89262 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:19.488569 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8feb3e26-7737-4f83-bc59-4ffaa7a89262 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:19.490088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8feb3e26-7737-4f83-bc59-4ffaa7a89262 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:19.499029 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1162/2322] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:19 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:55:23.357966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac45743-0b32-402d-abf6-0a879b6e6a3b service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:23.432233 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac45743-0b32-402d-abf6-0a879b6e6a3b service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:23.445056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603f5966-34e7-4cb2-b1a0-b8cb9543f598 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:23.445737 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603f5966-34e7-4cb2-b1a0-b8cb9543f598 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:23.446149 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603f5966-34e7-4cb2-b1a0-b8cb9543f598 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:23.446434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603f5966-34e7-4cb2-b1a0-b8cb9543f598 admin admin] PATH_INFO: `/v3/users/d297f4947ce64cad93940e04c6a13a00` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:23.447611 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603f5966-34e7-4cb2-b1a0-b8cb9543f598 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d297f4947ce64cad93940e04c6a13a00)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:23.449532 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603f5966-34e7-4cb2-b1a0-b8cb9543f598 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:23.646359 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-603f5966-34e7-4cb2-b1a0-b8cb9543f598 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd297f4947ce64cad93940e04c6a13a00'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:23.662693 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-603f5966-34e7-4cb2-b1a0-b8cb9543f598 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd297f4947ce64cad93940e04c6a13a00'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:23.779744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-603f5966-34e7-4cb2-b1a0-b8cb9543f598 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd297f4947ce64cad93940e04c6a13a00'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:23.780378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-603f5966-34e7-4cb2-b1a0-b8cb9543f598 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd297f4947ce64cad93940e04c6a13a00'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:23.789133 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-603f5966-34e7-4cb2-b1a0-b8cb9543f598 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd297f4947ce64cad93940e04c6a13a00'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:23.795715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1161/2323] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:55:23 2022] DELETE /identity/v3/users/d297f4947ce64cad93940e04c6a13a00 => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:23.800928 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c763f7e1-108c-4b45-b4e9-c249d42e4a82 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:23.801502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c763f7e1-108c-4b45-b4e9-c249d42e4a82 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:23.801694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c763f7e1-108c-4b45-b4e9-c249d42e4a82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:23.841856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-c763f7e1-108c-4b45-b4e9-c249d42e4a82 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:23.843131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c763f7e1-108c-4b45-b4e9-c249d42e4a82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:23.878929 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:23.943518 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1163/2324] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:55:23.955113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603f5966-34e7-4cb2-b1a0-b8cb9543f598 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:23.956695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-603f5966-34e7-4cb2-b1a0-b8cb9543f598 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:24.038413 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603f5966-34e7-4cb2-b1a0-b8cb9543f598 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:24.042736 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e7b598-6f3e-4a06-bb2c-ddc243d6bb59 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:24.043448 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e7b598-6f3e-4a06-bb2c-ddc243d6bb59 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:24.070060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e7b598-6f3e-4a06-bb2c-ddc243d6bb59 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:24.070060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e7b598-6f3e-4a06-bb2c-ddc243d6bb59 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:24.070060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e7b598-6f3e-4a06-bb2c-ddc243d6bb59 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:24.070060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e7b598-6f3e-4a06-bb2c-ddc243d6bb59 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:24.070060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1162/2325] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:55:25.896883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c763f7e1-108c-4b45-b4e9-c249d42e4a82 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:25.898378 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c763f7e1-108c-4b45-b4e9-c249d42e4a82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:25.910619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c763f7e1-108c-4b45-b4e9-c249d42e4a82 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:25.915365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7386b063-7794-4da9-a185-ad8efc126f6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:25.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7386b063-7794-4da9-a185-ad8efc126f6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:25.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7386b063-7794-4da9-a185-ad8efc126f6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:25.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7386b063-7794-4da9-a185-ad8efc126f6f admin admin] PATH_INFO: `/v3/projects/78c2e054024346ecab252245a8ece557` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:25.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7386b063-7794-4da9-a185-ad8efc126f6f admin admin] RBAC: Authorizing `identity:delete_project(project_id=78c2e054024346ecab252245a8ece557)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:25.923112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7386b063-7794-4da9-a185-ad8efc126f6f admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:26.027578 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-7386b063-7794-4da9-a185-ad8efc126f6f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '78c2e054024346ecab252245a8ece557', 'request_id': 'req-7386b063-7794-4da9-a185-ad8efc126f6f'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:26.129132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-7386b063-7794-4da9-a185-ad8efc126f6f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '78c2e054024346ecab252245a8ece557', 'request_id': 'req-7386b063-7794-4da9-a185-ad8efc126f6f'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:26.135697 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-7386b063-7794-4da9-a185-ad8efc126f6f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '78c2e054024346ecab252245a8ece557', 'request_id': 'req-7386b063-7794-4da9-a185-ad8efc126f6f'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:26.147726 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-7386b063-7794-4da9-a185-ad8efc126f6f admin admin] The token cache is being invalidate because project 78c2e054024346ecab252245a8ece557 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:55:26.148138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-7386b063-7794-4da9-a185-ad8efc126f6f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:26.149154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-7386b063-7794-4da9-a185-ad8efc126f6f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:26.151431 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1164/2326] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:55:25 2022] DELETE /identity/v3/projects/78c2e054024346ecab252245a8ece557 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:26.155455 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e7b598-6f3e-4a06-bb2c-ddc243d6bb59 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:26.160409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:26.166247 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e7b598-6f3e-4a06-bb2c-ddc243d6bb59 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:26.227460 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2854edd-ae15-4099-9db1-811e0c5c04e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:26.228180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2854edd-ae15-4099-9db1-811e0c5c04e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:26.228513 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2854edd-ae15-4099-9db1-811e0c5c04e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:26.228792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2854edd-ae15-4099-9db1-811e0c5c04e6 admin admin] PATH_INFO: `/v3/users/6b2893eb3df149409e051fba1b5388f3` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:26.230071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2854edd-ae15-4099-9db1-811e0c5c04e6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6b2893eb3df149409e051fba1b5388f3)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:26.232048 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2854edd-ae15-4099-9db1-811e0c5c04e6 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:26.459528 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a2854edd-ae15-4099-9db1-811e0c5c04e6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6b2893eb3df149409e051fba1b5388f3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:26.471725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a2854edd-ae15-4099-9db1-811e0c5c04e6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6b2893eb3df149409e051fba1b5388f3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:26.591313 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a2854edd-ae15-4099-9db1-811e0c5c04e6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6b2893eb3df149409e051fba1b5388f3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:26.591930 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a2854edd-ae15-4099-9db1-811e0c5c04e6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6b2893eb3df149409e051fba1b5388f3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:26.605579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a2854edd-ae15-4099-9db1-811e0c5c04e6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6b2893eb3df149409e051fba1b5388f3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:26.611791 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1163/2327] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:55:26 2022] DELETE /identity/v3/users/6b2893eb3df149409e051fba1b5388f3 => generated 0 bytes in 457 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:27.325413 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7386b063-7794-4da9-a185-ad8efc126f6f admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:27.327532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-7386b063-7794-4da9-a185-ad8efc126f6f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:27.342884 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7386b063-7794-4da9-a185-ad8efc126f6f admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:27.403016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d705cf20-e513-4c77-b4e9-2358a06572c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:27.403578 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d705cf20-e513-4c77-b4e9-2358a06572c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:27.403862 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d705cf20-e513-4c77-b4e9-2358a06572c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:27.404139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d705cf20-e513-4c77-b4e9-2358a06572c4 admin admin] PATH_INFO: `/v3/projects/14406d9a93fe450aa372e291225a5612` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:27.405266 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d705cf20-e513-4c77-b4e9-2358a06572c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=14406d9a93fe450aa372e291225a5612)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:27.407225 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d705cf20-e513-4c77-b4e9-2358a06572c4 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:27.530293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d705cf20-e513-4c77-b4e9-2358a06572c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '14406d9a93fe450aa372e291225a5612', 'request_id': 'req-d705cf20-e513-4c77-b4e9-2358a06572c4'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:27.602308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d705cf20-e513-4c77-b4e9-2358a06572c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '14406d9a93fe450aa372e291225a5612', 'request_id': 'req-d705cf20-e513-4c77-b4e9-2358a06572c4'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:27.603737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:27.608640 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d705cf20-e513-4c77-b4e9-2358a06572c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '14406d9a93fe450aa372e291225a5612', 'request_id': 'req-d705cf20-e513-4c77-b4e9-2358a06572c4'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:27.620945 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d705cf20-e513-4c77-b4e9-2358a06572c4 admin admin] The token cache is being invalidate because project 14406d9a93fe450aa372e291225a5612 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:55:27.621237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d705cf20-e513-4c77-b4e9-2358a06572c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:27.621891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-d705cf20-e513-4c77-b4e9-2358a06572c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:27.623479 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1165/2328] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:55:27 2022] DELETE /identity/v3/projects/14406d9a93fe450aa372e291225a5612 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:27.629034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55193932-5d7d-47fe-9e66-e5c5b9dc9135 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:27.629567 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55193932-5d7d-47fe-9e66-e5c5b9dc9135 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:27.629862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55193932-5d7d-47fe-9e66-e5c5b9dc9135 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:27.649632 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-55193932-5d7d-47fe-9e66-e5c5b9dc9135 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:27.651193 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-55193932-5d7d-47fe-9e66-e5c5b9dc9135 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:27.713783 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1164/2329] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:55:27.721898 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9cc473-3226-4346-95bd-72bc53d94a0f None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:27.722186 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9cc473-3226-4346-95bd-72bc53d94a0f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:27.722462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9cc473-3226-4346-95bd-72bc53d94a0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:27.745021 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-fa9cc473-3226-4346-95bd-72bc53d94a0f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:27.746144 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-fa9cc473-3226-4346-95bd-72bc53d94a0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:27.754197 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1166/2330] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:55:27.759348 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55193932-5d7d-47fe-9e66-e5c5b9dc9135 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:27.769341 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55193932-5d7d-47fe-9e66-e5c5b9dc9135 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:27.774492 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13977c57-cdac-4a87-b358-e4b8e26f3b19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:27.775204 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13977c57-cdac-4a87-b358-e4b8e26f3b19 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:27.775492 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13977c57-cdac-4a87-b358-e4b8e26f3b19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:27.775770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13977c57-cdac-4a87-b358-e4b8e26f3b19 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:27.776423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13977c57-cdac-4a87-b358-e4b8e26f3b19 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:55:27.776731 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13977c57-cdac-4a87-b358-e4b8e26f3b19 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:27.778316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13977c57-cdac-4a87-b358-e4b8e26f3b19 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:27.790551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1165/2331] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:55:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:55:27.795161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9cc473-3226-4346-95bd-72bc53d94a0f None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:27.804790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9cc473-3226-4346-95bd-72bc53d94a0f None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:27.809875 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9f0fbd-01a0-4435-b0da-2e06ac3a1326 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:27.810467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9f0fbd-01a0-4435-b0da-2e06ac3a1326 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:27.810742 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9f0fbd-01a0-4435-b0da-2e06ac3a1326 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:27.811051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9f0fbd-01a0-4435-b0da-2e06ac3a1326 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:27.811688 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9f0fbd-01a0-4435-b0da-2e06ac3a1326 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:27.813565 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9f0fbd-01a0-4435-b0da-2e06ac3a1326 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:27.919450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1167/2332] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:55:27 2022] POST /identity/v3/projects => generated 389 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:55:27.923814 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13977c57-cdac-4a87-b358-e4b8e26f3b19 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:27.935536 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13977c57-cdac-4a87-b358-e4b8e26f3b19 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:27.940279 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:27.995397 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf26317-d71e-4abd-a402-bd5a2fa62f55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:27.996043 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf26317-d71e-4abd-a402-bd5a2fa62f55 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:27.996809 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf26317-d71e-4abd-a402-bd5a2fa62f55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:27.996809 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf26317-d71e-4abd-a402-bd5a2fa62f55 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:27.997286 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf26317-d71e-4abd-a402-bd5a2fa62f55 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:27.999325 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf26317-d71e-4abd-a402-bd5a2fa62f55 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:28.069889 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1166/2333] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:55:27 2022] POST /identity/v3/users => generated 295 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:55:28.070668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9f0fbd-01a0-4435-b0da-2e06ac3a1326 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:28.082073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9f0fbd-01a0-4435-b0da-2e06ac3a1326 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:28.088063 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caec3b0b-968b-4f9c-a00c-0e8c5a286713 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:28.088771 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caec3b0b-968b-4f9c-a00c-0e8c5a286713 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:28.089151 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caec3b0b-968b-4f9c-a00c-0e8c5a286713 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:28.089549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caec3b0b-968b-4f9c-a00c-0e8c5a286713 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:28.090261 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caec3b0b-968b-4f9c-a00c-0e8c5a286713 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:28.091978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caec3b0b-968b-4f9c-a00c-0e8c5a286713 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:28.104049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1168/2334] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:55:28 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:55:28.108226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf26317-d71e-4abd-a402-bd5a2fa62f55 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:28.118553 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf26317-d71e-4abd-a402-bd5a2fa62f55 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:28.123440 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b66d21-be19-4880-884c-3ccc4e42ff85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:28.124048 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b66d21-be19-4880-884c-3ccc4e42ff85 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:28.124334 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b66d21-be19-4880-884c-3ccc4e42ff85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:28.124613 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b66d21-be19-4880-884c-3ccc4e42ff85 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:28.125256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b66d21-be19-4880-884c-3ccc4e42ff85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:28.126880 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b66d21-be19-4880-884c-3ccc4e42ff85 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:28.138331 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1167/2335] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:55:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:55:28.142729 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caec3b0b-968b-4f9c-a00c-0e8c5a286713 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:28.153447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caec3b0b-968b-4f9c-a00c-0e8c5a286713 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:28.158642 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6232d0-f765-4c1b-a25f-91709875d98f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:28.159565 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6232d0-f765-4c1b-a25f-91709875d98f admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:28.160010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6232d0-f765-4c1b-a25f-91709875d98f admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:28.160371 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6232d0-f765-4c1b-a25f-91709875d98f admin admin] PATH_INFO: `/v3/projects/8b3826303daa4e75a5c7214fca6066b2/users/341f27d5485d413396c7e81865a9ad77/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:28.183604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6232d0-f765-4c1b-a25f-91709875d98f admin admin] RBAC: Authorizing `identity:create_grant(project_id=8b3826303daa4e75a5c7214fca6066b2, user_id=341f27d5485d413396c7e81865a9ad77, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:28.188118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6232d0-f765-4c1b-a25f-91709875d98f admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:28.290194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1169/2336] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:55:28 2022] PUT /identity/v3/projects/8b3826303daa4e75a5c7214fca6066b2/users/341f27d5485d413396c7e81865a9ad77/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:28.301568 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14768c89-4fb4-44dd-98c8-3fbd49f5da1d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:28.302039 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14768c89-4fb4-44dd-98c8-3fbd49f5da1d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:28.302523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14768c89-4fb4-44dd-98c8-3fbd49f5da1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:28.350976 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-14768c89-4fb4-44dd-98c8-3fbd49f5da1d None None] MFA Rules not processed for user `341f27d5485d413396c7e81865a9ad77`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:28.355605 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-14768c89-4fb4-44dd-98c8-3fbd49f5da1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:28.465089 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1168/2337] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:28 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:55:28.469372 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6232d0-f765-4c1b-a25f-91709875d98f admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:28.591780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6232d0-f765-4c1b-a25f-91709875d98f admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:28.591780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc27b04-f5d8-4842-b4df-094a58d9bb9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:28.591780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc27b04-f5d8-4842-b4df-094a58d9bb9b admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:28.591780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc27b04-f5d8-4842-b4df-094a58d9bb9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:28.592352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc27b04-f5d8-4842-b4df-094a58d9bb9b admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:28.592352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc27b04-f5d8-4842-b4df-094a58d9bb9b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:28.592352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc27b04-f5d8-4842-b4df-094a58d9bb9b admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:28.749588 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1170/2338] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:55:28 2022] POST /identity/v3/projects => generated 387 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:55:28.754219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14768c89-4fb4-44dd-98c8-3fbd49f5da1d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:28.770135 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14768c89-4fb4-44dd-98c8-3fbd49f5da1d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:28.848471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1f1f94-417f-4cbb-a4ae-c6e19c9afd4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:28.849778 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1f1f94-417f-4cbb-a4ae-c6e19c9afd4f admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:28.850134 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1f1f94-417f-4cbb-a4ae-c6e19c9afd4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:28.850486 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1f1f94-417f-4cbb-a4ae-c6e19c9afd4f admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:28.851247 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1f1f94-417f-4cbb-a4ae-c6e19c9afd4f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:28.853266 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1f1f94-417f-4cbb-a4ae-c6e19c9afd4f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:29.073274 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1169/2339] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:55:28 2022] POST /identity/v3/users => generated 300 bytes in 320 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:55:29.077411 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc27b04-f5d8-4842-b4df-094a58d9bb9b admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:29.088314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc27b04-f5d8-4842-b4df-094a58d9bb9b admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:29.092834 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b36f79-3bb1-4303-918d-5dd4081a4f7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:29.093430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b36f79-3bb1-4303-918d-5dd4081a4f7b admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:29.093712 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b36f79-3bb1-4303-918d-5dd4081a4f7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:29.093985 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b36f79-3bb1-4303-918d-5dd4081a4f7b admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:29.094618 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b36f79-3bb1-4303-918d-5dd4081a4f7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:29.096243 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b36f79-3bb1-4303-918d-5dd4081a4f7b admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:29.107754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1171/2340] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:55:29 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:55:29.111866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1f1f94-417f-4cbb-a4ae-c6e19c9afd4f admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:29.122240 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1f1f94-417f-4cbb-a4ae-c6e19c9afd4f admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:29.126984 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6111cb-579b-424b-b356-a32c23ef4e1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:29.127648 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6111cb-579b-424b-b356-a32c23ef4e1d admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:29.128100 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6111cb-579b-424b-b356-a32c23ef4e1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:29.128347 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6111cb-579b-424b-b356-a32c23ef4e1d admin admin] PATH_INFO: `/v3/projects/e0e968190cc04f658ca5220abeb9c7ea/users/da1583a09b3b41e0be390325e6bbfb0a/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:29.131475 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:29.145259 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf6111cb-579b-424b-b356-a32c23ef4e1d admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0e968190cc04f658ca5220abeb9c7ea, user_id=da1583a09b3b41e0be390325e6bbfb0a, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:29.149103 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf6111cb-579b-424b-b356-a32c23ef4e1d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:29.272334 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1170/2341] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:55:29 2022] PUT /identity/v3/projects/e0e968190cc04f658ca5220abeb9c7ea/users/da1583a09b3b41e0be390325e6bbfb0a/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:29.278123 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b36f79-3bb1-4303-918d-5dd4081a4f7b admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:29.286555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b36f79-3bb1-4303-918d-5dd4081a4f7b admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:29.367110 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:DELETE=2,keystone:SELECT=95,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:29.374424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f7d53c-21d2-44f7-affd-0497eb1b401a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:29.375072 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f7d53c-21d2-44f7-affd-0497eb1b401a admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:29.375352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f7d53c-21d2-44f7-affd-0497eb1b401a admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:29.375626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f7d53c-21d2-44f7-affd-0497eb1b401a admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:29.376270 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f7d53c-21d2-44f7-affd-0497eb1b401a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:29.377856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f7d53c-21d2-44f7-affd-0497eb1b401a admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:29.390951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1172/2342] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:55:29 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:55:29.394670 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6111cb-579b-424b-b356-a32c23ef4e1d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:29.406503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6111cb-579b-424b-b356-a32c23ef4e1d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:29.412484 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a47102a-189a-4ba5-952c-7fcd8923f806 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:29.414531 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a47102a-189a-4ba5-952c-7fcd8923f806 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:29.414531 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a47102a-189a-4ba5-952c-7fcd8923f806 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:29.414531 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a47102a-189a-4ba5-952c-7fcd8923f806 admin admin] PATH_INFO: `/v3/projects/e0e968190cc04f658ca5220abeb9c7ea/users/da1583a09b3b41e0be390325e6bbfb0a/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:29.415829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a47102a-189a-4ba5-952c-7fcd8923f806 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0e968190cc04f658ca5220abeb9c7ea, user_id=da1583a09b3b41e0be390325e6bbfb0a, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:29.418687 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a47102a-189a-4ba5-952c-7fcd8923f806 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:29.431394 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1171/2343] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:55:29 2022] PUT /identity/v3/projects/e0e968190cc04f658ca5220abeb9c7ea/users/da1583a09b3b41e0be390325e6bbfb0a/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:29.444198 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb9c375-d95d-4a59-8662-50a790d23451 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:29.444484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb9c375-d95d-4a59-8662-50a790d23451 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:29.444757 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb9c375-d95d-4a59-8662-50a790d23451 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:29.480478 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-fcb9c375-d95d-4a59-8662-50a790d23451 None None] MFA Rules not processed for user `da1583a09b3b41e0be390325e6bbfb0a`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:29.481748 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-fcb9c375-d95d-4a59-8662-50a790d23451 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:29.596218 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1173/2344] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:29 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:55:29.609072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a47102a-189a-4ba5-952c-7fcd8923f806 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:29.610924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5a47102a-189a-4ba5-952c-7fcd8923f806 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:29.690850 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a47102a-189a-4ba5-952c-7fcd8923f806 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:29.696955 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f58cbeb-8bcc-4855-b0ad-6901b313df22 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:29.696955 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f58cbeb-8bcc-4855-b0ad-6901b313df22 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:29.696955 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f58cbeb-8bcc-4855-b0ad-6901b313df22 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:29.696955 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f58cbeb-8bcc-4855-b0ad-6901b313df22 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:29.706728 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f58cbeb-8bcc-4855-b0ad-6901b313df22 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:29.708939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f58cbeb-8bcc-4855-b0ad-6901b313df22 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:29.719501 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1172/2345] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:29 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:55:29.878461 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb9c375-d95d-4a59-8662-50a790d23451 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:29.950338 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb9c375-d95d-4a59-8662-50a790d23451 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:29.955076 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1cb1d8-ee01-455e-ba3f-3ac14d67f9d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:29.955713 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1cb1d8-ee01-455e-ba3f-3ac14d67f9d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:29.956009 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1cb1d8-ee01-455e-ba3f-3ac14d67f9d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:29.956267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1cb1d8-ee01-455e-ba3f-3ac14d67f9d4 admin admin] PATH_INFO: `/v3/users/341f27d5485d413396c7e81865a9ad77` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:29.957382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1cb1d8-ee01-455e-ba3f-3ac14d67f9d4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=341f27d5485d413396c7e81865a9ad77)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:29.959526 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1cb1d8-ee01-455e-ba3f-3ac14d67f9d4 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:30.305341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-ca1cb1d8-ee01-455e-ba3f-3ac14d67f9d4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '341f27d5485d413396c7e81865a9ad77'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:30.456400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-ca1cb1d8-ee01-455e-ba3f-3ac14d67f9d4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '341f27d5485d413396c7e81865a9ad77'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:30.464488 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-ca1cb1d8-ee01-455e-ba3f-3ac14d67f9d4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '341f27d5485d413396c7e81865a9ad77'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:30.465382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-ca1cb1d8-ee01-455e-ba3f-3ac14d67f9d4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '341f27d5485d413396c7e81865a9ad77'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:30.475415 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:30.478151 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-ca1cb1d8-ee01-455e-ba3f-3ac14d67f9d4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '341f27d5485d413396c7e81865a9ad77'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:30.485080 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1174/2346] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:55:29 2022] DELETE /identity/v3/users/341f27d5485d413396c7e81865a9ad77 => generated 0 bytes in 607 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:30.489233 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905eb3dc-6270-489f-847b-60e756304068 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:30.489950 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905eb3dc-6270-489f-847b-60e756304068 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:30.489950 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905eb3dc-6270-489f-847b-60e756304068 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:30.510014 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-905eb3dc-6270-489f-847b-60e756304068 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:30.511746 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-905eb3dc-6270-489f-847b-60e756304068 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:30.607671 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1173/2347] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:55:30.618380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1cb1d8-ee01-455e-ba3f-3ac14d67f9d4 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:30.620119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ca1cb1d8-ee01-455e-ba3f-3ac14d67f9d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:30.681083 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1cb1d8-ee01-455e-ba3f-3ac14d67f9d4 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:30.684890 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f1183d-0789-4b73-b7eb-8a65bc1f4f71 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:30.685490 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f1183d-0789-4b73-b7eb-8a65bc1f4f71 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:30.685766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f1183d-0789-4b73-b7eb-8a65bc1f4f71 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:30.686129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f1183d-0789-4b73-b7eb-8a65bc1f4f71 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:30.699817 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f1183d-0789-4b73-b7eb-8a65bc1f4f71 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:30.701784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f1183d-0789-4b73-b7eb-8a65bc1f4f71 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:30.721338 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1175/2348] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:55:31.199557 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905eb3dc-6270-489f-847b-60e756304068 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:31.201133 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-905eb3dc-6270-489f-847b-60e756304068 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:31.214256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905eb3dc-6270-489f-847b-60e756304068 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:31.219110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ab9ed5-f76a-4d2f-8045-63ce6afe0552 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:31.219752 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ab9ed5-f76a-4d2f-8045-63ce6afe0552 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:31.222709 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ab9ed5-f76a-4d2f-8045-63ce6afe0552 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:31.222709 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ab9ed5-f76a-4d2f-8045-63ce6afe0552 admin admin] PATH_INFO: `/v3/projects/8b3826303daa4e75a5c7214fca6066b2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:31.222709 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ab9ed5-f76a-4d2f-8045-63ce6afe0552 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8b3826303daa4e75a5c7214fca6066b2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:31.224461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ab9ed5-f76a-4d2f-8045-63ce6afe0552 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:31.422454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-28ab9ed5-f76a-4d2f-8045-63ce6afe0552 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8b3826303daa4e75a5c7214fca6066b2', 'request_id': 'req-28ab9ed5-f76a-4d2f-8045-63ce6afe0552'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:31.430280 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-28ab9ed5-f76a-4d2f-8045-63ce6afe0552 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8b3826303daa4e75a5c7214fca6066b2', 'request_id': 'req-28ab9ed5-f76a-4d2f-8045-63ce6afe0552'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:31.527908 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-28ab9ed5-f76a-4d2f-8045-63ce6afe0552 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8b3826303daa4e75a5c7214fca6066b2', 'request_id': 'req-28ab9ed5-f76a-4d2f-8045-63ce6afe0552'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:31.535485 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:31.542825 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-28ab9ed5-f76a-4d2f-8045-63ce6afe0552 admin admin] The token cache is being invalidate because project 8b3826303daa4e75a5c7214fca6066b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:55:31.543220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-28ab9ed5-f76a-4d2f-8045-63ce6afe0552 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:31.543933 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-28ab9ed5-f76a-4d2f-8045-63ce6afe0552 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:31.545444 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1174/2349] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:55:31 2022] DELETE /identity/v3/projects/8b3826303daa4e75a5c7214fca6066b2 => generated 0 bytes in 346 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:31.549410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f1183d-0789-4b73-b7eb-8a65bc1f4f71 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:31.559720 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f1183d-0789-4b73-b7eb-8a65bc1f4f71 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:31.617805 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40adc941-002a-4065-a96b-02fce55599d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:31.618673 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40adc941-002a-4065-a96b-02fce55599d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:31.618850 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40adc941-002a-4065-a96b-02fce55599d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:31.619203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40adc941-002a-4065-a96b-02fce55599d3 admin admin] PATH_INFO: `/v3/users/da1583a09b3b41e0be390325e6bbfb0a` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:31.620343 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40adc941-002a-4065-a96b-02fce55599d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=da1583a09b3b41e0be390325e6bbfb0a)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:31.622308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40adc941-002a-4065-a96b-02fce55599d3 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:31.857052 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-40adc941-002a-4065-a96b-02fce55599d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'da1583a09b3b41e0be390325e6bbfb0a'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:31.928758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-40adc941-002a-4065-a96b-02fce55599d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'da1583a09b3b41e0be390325e6bbfb0a'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:31.942540 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-40adc941-002a-4065-a96b-02fce55599d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'da1583a09b3b41e0be390325e6bbfb0a'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:31.943268 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-40adc941-002a-4065-a96b-02fce55599d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'da1583a09b3b41e0be390325e6bbfb0a'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:31.950606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-40adc941-002a-4065-a96b-02fce55599d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'da1583a09b3b41e0be390325e6bbfb0a'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:31.965515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1176/2350] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:55:31 2022] DELETE /identity/v3/users/da1583a09b3b41e0be390325e6bbfb0a => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:32.349500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ab9ed5-f76a-4d2f-8045-63ce6afe0552 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:32.351116 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-28ab9ed5-f76a-4d2f-8045-63ce6afe0552 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:32.408828 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ab9ed5-f76a-4d2f-8045-63ce6afe0552 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:32.433753 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40adc941-002a-4065-a96b-02fce55599d3 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:32.435240 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-40adc941-002a-4065-a96b-02fce55599d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:32.477723 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:32.499188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff5a2e0-0853-43ef-833b-bd4bdfbc1e59 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:32.499188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff5a2e0-0853-43ef-833b-bd4bdfbc1e59 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:32.499188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff5a2e0-0853-43ef-833b-bd4bdfbc1e59 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:32.499188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff5a2e0-0853-43ef-833b-bd4bdfbc1e59 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:32.499617 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9ff5a2e0-0853-43ef-833b-bd4bdfbc1e59 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:32.527543 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40adc941-002a-4065-a96b-02fce55599d3 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:32.532290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb166b6-b669-4b10-b075-7fcad680141e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:32.532930 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb166b6-b669-4b10-b075-7fcad680141e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:32.533211 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb166b6-b669-4b10-b075-7fcad680141e admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:32.533544 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb166b6-b669-4b10-b075-7fcad680141e admin admin] PATH_INFO: `/v3/projects/e0e968190cc04f658ca5220abeb9c7ea` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:32.534653 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb166b6-b669-4b10-b075-7fcad680141e admin admin] RBAC: Authorizing `identity:delete_project(project_id=e0e968190cc04f658ca5220abeb9c7ea)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:32.537348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb166b6-b669-4b10-b075-7fcad680141e admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:32.571010 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff5a2e0-0853-43ef-833b-bd4bdfbc1e59 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:32.572476 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff5a2e0-0853-43ef-833b-bd4bdfbc1e59 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:32.584497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1175/2351] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:55:32.642752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-4eb166b6-b669-4b10-b075-7fcad680141e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e0e968190cc04f658ca5220abeb9c7ea', 'request_id': 'req-4eb166b6-b669-4b10-b075-7fcad680141e'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:32.756806 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-4eb166b6-b669-4b10-b075-7fcad680141e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e0e968190cc04f658ca5220abeb9c7ea', 'request_id': 'req-4eb166b6-b669-4b10-b075-7fcad680141e'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:32.763204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-4eb166b6-b669-4b10-b075-7fcad680141e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e0e968190cc04f658ca5220abeb9c7ea', 'request_id': 'req-4eb166b6-b669-4b10-b075-7fcad680141e'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:32.776086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-4eb166b6-b669-4b10-b075-7fcad680141e admin admin] The token cache is being invalidate because project e0e968190cc04f658ca5220abeb9c7ea was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:55:32.776459 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-4eb166b6-b669-4b10-b075-7fcad680141e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:32.777129 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-4eb166b6-b669-4b10-b075-7fcad680141e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:32.782286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1177/2352] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:55:32 2022] DELETE /identity/v3/projects/e0e968190cc04f658ca5220abeb9c7ea => generated 0 bytes in 346 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:32.784114 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c61a1f9-2389-4e10-8b63-ee63a5c97a6b None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:32.784347 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c61a1f9-2389-4e10-8b63-ee63a5c97a6b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:32.784649 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c61a1f9-2389-4e10-8b63-ee63a5c97a6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:32.806574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7c61a1f9-2389-4e10-8b63-ee63a5c97a6b None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:32.806574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7c61a1f9-2389-4e10-8b63-ee63a5c97a6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:32.876804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1176/2353] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:55:32.885487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd210112-611c-40c2-8a39-3df2a695944e None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:32.886439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd210112-611c-40c2-8a39-3df2a695944e None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:32.886933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd210112-611c-40c2-8a39-3df2a695944e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:32.909355 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-dd210112-611c-40c2-8a39-3df2a695944e None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:32.910744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-dd210112-611c-40c2-8a39-3df2a695944e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:32.919361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1178/2354] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:55:32.923895 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c61a1f9-2389-4e10-8b63-ee63a5c97a6b None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:32.929081 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:32.934674 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c61a1f9-2389-4e10-8b63-ee63a5c97a6b None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:32.939790 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d652fd6-4d84-4649-b648-cf51b3d58f5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:32.940390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d652fd6-4d84-4649-b648-cf51b3d58f5b admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:32.940667 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d652fd6-4d84-4649-b648-cf51b3d58f5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:32.940939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d652fd6-4d84-4649-b648-cf51b3d58f5b admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:32.941577 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d652fd6-4d84-4649-b648-cf51b3d58f5b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:55:32.941877 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d652fd6-4d84-4649-b648-cf51b3d58f5b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:32.944042 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d652fd6-4d84-4649-b648-cf51b3d58f5b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:32.955722 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1177/2355] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:55:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:55:32.961211 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd210112-611c-40c2-8a39-3df2a695944e None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:32.971883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd210112-611c-40c2-8a39-3df2a695944e None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:32.976651 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facf2916-771a-45bc-8e1c-3a3b01c9c7e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:32.977284 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facf2916-771a-45bc-8e1c-3a3b01c9c7e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:32.977649 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facf2916-771a-45bc-8e1c-3a3b01c9c7e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:32.977935 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facf2916-771a-45bc-8e1c-3a3b01c9c7e6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:32.978630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facf2916-771a-45bc-8e1c-3a3b01c9c7e6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:32.980673 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facf2916-771a-45bc-8e1c-3a3b01c9c7e6 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:33.067817 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1179/2356] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:55:32 2022] POST /identity/v3/projects => generated 385 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:55:33.071811 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d652fd6-4d84-4649-b648-cf51b3d58f5b admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:33.082395 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d652fd6-4d84-4649-b648-cf51b3d58f5b admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:33.150329 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157aede2-11f9-497d-9fd3-8fa33418f5bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:33.151040 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157aede2-11f9-497d-9fd3-8fa33418f5bb admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:33.151323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157aede2-11f9-497d-9fd3-8fa33418f5bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:33.151596 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157aede2-11f9-497d-9fd3-8fa33418f5bb admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:33.152278 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157aede2-11f9-497d-9fd3-8fa33418f5bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:33.154370 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157aede2-11f9-497d-9fd3-8fa33418f5bb admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:33.253348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f536da4a-d124-49e8-a986-69f8617b262f None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:33.253721 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f536da4a-d124-49e8-a986-69f8617b262f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:33.254025 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f536da4a-d124-49e8-a986-69f8617b262f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:33.276557 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-f536da4a-d124-49e8-a986-69f8617b262f None None] MFA Rules not processed for user `d40810f61ead4c3da414194b325da80a`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:33.277749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-f536da4a-d124-49e8-a986-69f8617b262f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:33.306321 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1178/2357] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:55:33 2022] POST /identity/v3/users => generated 293 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:55:33.310527 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157aede2-11f9-497d-9fd3-8fa33418f5bb admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:33.322473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157aede2-11f9-497d-9fd3-8fa33418f5bb admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:33.328041 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a170545-271b-494f-8746-4384b93a95bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:33.328884 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a170545-271b-494f-8746-4384b93a95bb admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:33.329225 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a170545-271b-494f-8746-4384b93a95bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:33.329440 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a170545-271b-494f-8746-4384b93a95bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:33.330501 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a170545-271b-494f-8746-4384b93a95bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:33.331720 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a170545-271b-494f-8746-4384b93a95bb admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:33.344328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1179/2358] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:55:33 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:55:33.348022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a170545-271b-494f-8746-4384b93a95bb admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:33.353250 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1180/2359] 10.222.0.49 () {58 vars in 967 bytes} [Fri Sep 16 03:55:33 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:55:33.358475 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a170545-271b-494f-8746-4384b93a95bb admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:33.363616 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecaeec9a-1c78-4686-a82a-9aa41a31f782 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:33.364859 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaeec9a-1c78-4686-a82a-9aa41a31f782 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:33.365135 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaeec9a-1c78-4686-a82a-9aa41a31f782 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:33.365405 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaeec9a-1c78-4686-a82a-9aa41a31f782 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:33.366040 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecaeec9a-1c78-4686-a82a-9aa41a31f782 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:33.372404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecaeec9a-1c78-4686-a82a-9aa41a31f782 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:33.378254 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1180/2360] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:55:33 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:55:33.382746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f536da4a-d124-49e8-a986-69f8617b262f None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:33.392114 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecaeec9a-1c78-4686-a82a-9aa41a31f782 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:33.393887 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ecaeec9a-1c78-4686-a82a-9aa41a31f782 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:33.399118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f536da4a-d124-49e8-a986-69f8617b262f None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:33.404958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06c50c3-53eb-4fd3-a6c0-df8264f52ae6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:33.405625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06c50c3-53eb-4fd3-a6c0-df8264f52ae6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:33.405903 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06c50c3-53eb-4fd3-a6c0-df8264f52ae6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:33.406240 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06c50c3-53eb-4fd3-a6c0-df8264f52ae6 admin admin] PATH_INFO: `/v3/projects/1da1b7daef124a18bba699cd70b10e62/users/939b3bedc9b34751899c7cee609168ef/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:33.422645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06c50c3-53eb-4fd3-a6c0-df8264f52ae6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1da1b7daef124a18bba699cd70b10e62, user_id=939b3bedc9b34751899c7cee609168ef, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:33.425450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06c50c3-53eb-4fd3-a6c0-df8264f52ae6 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:33.481487 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecaeec9a-1c78-4686-a82a-9aa41a31f782 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:33.486649 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac9ea80-18fe-4569-a845-dd3e292d54be service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa30da9972e140d7905c522576681429', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:33.487292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac9ea80-18fe-4569-a845-dd3e292d54be service swift] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:33.487693 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac9ea80-18fe-4569-a845-dd3e292d54be service swift] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:33.487981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac9ea80-18fe-4569-a845-dd3e292d54be service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:33.496674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:33.498271 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac9ea80-18fe-4569-a845-dd3e292d54be service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:33.500090 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac9ea80-18fe-4569-a845-dd3e292d54be service swift] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:33.512617 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1181/2362] 10.222.0.49 () {60 vars in 1384 bytes} [Fri Sep 16 03:55:33 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 16 03:55:33.522224 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f67467f-9f25-4d2c-b5e8-5abaae447724 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:33.522550 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f67467f-9f25-4d2c-b5e8-5abaae447724 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:33.523180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f67467f-9f25-4d2c-b5e8-5abaae447724 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:33.533949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1181/2362] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:55:33 2022] PUT /identity/v3/projects/1da1b7daef124a18bba699cd70b10e62/users/939b3bedc9b34751899c7cee609168ef/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:33.571623 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1f67467f-9f25-4d2c-b5e8-5abaae447724 None None] MFA Rules not processed for user `939b3bedc9b34751899c7cee609168ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:33.578544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1f67467f-9f25-4d2c-b5e8-5abaae447724 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:33.617024 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:33.653627 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1182/2363] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:33 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:55:33.658106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06c50c3-53eb-4fd3-a6c0-df8264f52ae6 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:33.733917 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06c50c3-53eb-4fd3-a6c0-df8264f52ae6 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:33.738786 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fb40a7-a6d8-4d09-9cbc-3da9e3d76239 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:33.739722 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fb40a7-a6d8-4d09-9cbc-3da9e3d76239 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:33.740195 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fb40a7-a6d8-4d09-9cbc-3da9e3d76239 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:33.740473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fb40a7-a6d8-4d09-9cbc-3da9e3d76239 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:33.741128 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fb40a7-a6d8-4d09-9cbc-3da9e3d76239 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:33.743277 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fb40a7-a6d8-4d09-9cbc-3da9e3d76239 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:33.921270 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1182/2364] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:55:33 2022] POST /identity/v3/projects => generated 385 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:55:33.926272 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f67467f-9f25-4d2c-b5e8-5abaae447724 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:33.939428 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f67467f-9f25-4d2c-b5e8-5abaae447724 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:34.057228 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef38dd4-1b62-4eff-b6e6-aeee82ea5684 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:34.057228 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef38dd4-1b62-4eff-b6e6-aeee82ea5684 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:34.057228 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef38dd4-1b62-4eff-b6e6-aeee82ea5684 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:34.057228 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef38dd4-1b62-4eff-b6e6-aeee82ea5684 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:34.058429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef38dd4-1b62-4eff-b6e6-aeee82ea5684 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:34.058429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef38dd4-1b62-4eff-b6e6-aeee82ea5684 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:34.198376 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1183/2365] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:55:33 2022] POST /identity/v3/users => generated 299 bytes in 273 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:55:34.202703 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fb40a7-a6d8-4d09-9cbc-3da9e3d76239 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:34.215692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fb40a7-a6d8-4d09-9cbc-3da9e3d76239 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:34.225517 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eff48a7-0cd7-43b4-a875-39c2bd317a1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:34.227527 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eff48a7-0cd7-43b4-a875-39c2bd317a1c admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:34.227813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eff48a7-0cd7-43b4-a875-39c2bd317a1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:34.228090 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eff48a7-0cd7-43b4-a875-39c2bd317a1c admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:34.229136 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eff48a7-0cd7-43b4-a875-39c2bd317a1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:34.231310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eff48a7-0cd7-43b4-a875-39c2bd317a1c admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:34.246094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1183/2366] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:55:34 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:55:34.250290 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef38dd4-1b62-4eff-b6e6-aeee82ea5684 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:34.261051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef38dd4-1b62-4eff-b6e6-aeee82ea5684 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:34.265649 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c77d72b-ea6d-4b78-ae97-e0895468e89d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:34.266297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c77d72b-ea6d-4b78-ae97-e0895468e89d admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:34.266574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c77d72b-ea6d-4b78-ae97-e0895468e89d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:34.266848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c77d72b-ea6d-4b78-ae97-e0895468e89d admin admin] PATH_INFO: `/v3/projects/fd04ba91ce40452abed64ba2516e3333/users/0a982d07f9b04649b841c271a4f14174/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:34.282726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c77d72b-ea6d-4b78-ae97-e0895468e89d admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd04ba91ce40452abed64ba2516e3333, user_id=0a982d07f9b04649b841c271a4f14174, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:34.285409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c77d72b-ea6d-4b78-ae97-e0895468e89d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:34.377904 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1184/2367] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:55:34 2022] PUT /identity/v3/projects/fd04ba91ce40452abed64ba2516e3333/users/0a982d07f9b04649b841c271a4f14174/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:34.380345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eff48a7-0cd7-43b4-a875-39c2bd317a1c admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:34.395364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eff48a7-0cd7-43b4-a875-39c2bd317a1c admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:34.503638 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a944ed-0c7b-4df9-bd37-c0815cba7f09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:34.504529 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a944ed-0c7b-4df9-bd37-c0815cba7f09 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:34.504837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a944ed-0c7b-4df9-bd37-c0815cba7f09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:34.505108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a944ed-0c7b-4df9-bd37-c0815cba7f09 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:34.505755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a944ed-0c7b-4df9-bd37-c0815cba7f09 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:34.510207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a944ed-0c7b-4df9-bd37-c0815cba7f09 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:34.563241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1184/2368] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:55:34 2022] GET /identity/v3/roles => generated 1420 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:55:34.566815 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c77d72b-ea6d-4b78-ae97-e0895468e89d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:34.584841 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c77d72b-ea6d-4b78-ae97-e0895468e89d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:34.594663 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93127611-21b5-49e0-9cdb-59f2c1cfcc4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:34.595438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93127611-21b5-49e0-9cdb-59f2c1cfcc4f admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:34.595725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93127611-21b5-49e0-9cdb-59f2c1cfcc4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:34.596002 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93127611-21b5-49e0-9cdb-59f2c1cfcc4f admin admin] PATH_INFO: `/v3/projects/fd04ba91ce40452abed64ba2516e3333/users/0a982d07f9b04649b841c271a4f14174/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:34.599586 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93127611-21b5-49e0-9cdb-59f2c1cfcc4f admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd04ba91ce40452abed64ba2516e3333, user_id=0a982d07f9b04649b841c271a4f14174, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:34.602785 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93127611-21b5-49e0-9cdb-59f2c1cfcc4f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:34.619596 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1185/2369] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:55:34 2022] PUT /identity/v3/projects/fd04ba91ce40452abed64ba2516e3333/users/0a982d07f9b04649b841c271a4f14174/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:34.631359 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4946bed2-c4a7-4ccd-9471-d11cd379a70e None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:34.631676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4946bed2-c4a7-4ccd-9471-d11cd379a70e None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:34.632030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4946bed2-c4a7-4ccd-9471-d11cd379a70e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:34.665483 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-4946bed2-c4a7-4ccd-9471-d11cd379a70e None None] MFA Rules not processed for user `0a982d07f9b04649b841c271a4f14174`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:34.667111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-4946bed2-c4a7-4ccd-9471-d11cd379a70e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:34.758290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:34.775147 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1185/2370] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:34 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:55:34.787039 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93127611-21b5-49e0-9cdb-59f2c1cfcc4f admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:34.787039 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-93127611-21b5-49e0-9cdb-59f2c1cfcc4f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:34.893820 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93127611-21b5-49e0-9cdb-59f2c1cfcc4f admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:34.899791 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49edb722-baaf-4002-accd-866e9e57cd18 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:34.901031 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49edb722-baaf-4002-accd-866e9e57cd18 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:34.901328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49edb722-baaf-4002-accd-866e9e57cd18 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:34.901617 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49edb722-baaf-4002-accd-866e9e57cd18 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:34.955521 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:34.979839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49edb722-baaf-4002-accd-866e9e57cd18 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:34.981347 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49edb722-baaf-4002-accd-866e9e57cd18 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:34.993564 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1186/2371] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:34 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 211 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:55:37.203489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4946bed2-c4a7-4ccd-9471-d11cd379a70e None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:37.206863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-4946bed2-c4a7-4ccd-9471-d11cd379a70e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:37.291543 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4946bed2-c4a7-4ccd-9471-d11cd379a70e None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:37.296866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17a151c-b01b-4ba3-a41d-66cbc4fefb87 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '176da4679fcb46148da9289fccdad5c2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:37.297501 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17a151c-b01b-4ba3-a41d-66cbc4fefb87 service placement] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:37.297774 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17a151c-b01b-4ba3-a41d-66cbc4fefb87 service placement] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:37.298047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17a151c-b01b-4ba3-a41d-66cbc4fefb87 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:37.299798 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d17a151c-b01b-4ba3-a41d-66cbc4fefb87 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:37.315776 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17a151c-b01b-4ba3-a41d-66cbc4fefb87 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:37.318305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17a151c-b01b-4ba3-a41d-66cbc4fefb87 service placement] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:37.330038 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1186/2372] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:37 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:55:40.943522 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49edb722-baaf-4002-accd-866e9e57cd18 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:40.945514 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-49edb722-baaf-4002-accd-866e9e57cd18 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:40.956731 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49edb722-baaf-4002-accd-866e9e57cd18 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:40.960729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656f111c-7835-4fe8-9a25-6da91474f591 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:40.961340 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656f111c-7835-4fe8-9a25-6da91474f591 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:40.961805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656f111c-7835-4fe8-9a25-6da91474f591 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:40.962084 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656f111c-7835-4fe8-9a25-6da91474f591 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:40.972783 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656f111c-7835-4fe8-9a25-6da91474f591 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:40.974254 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656f111c-7835-4fe8-9a25-6da91474f591 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:40.985079 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1187/2373] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:40 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:55:41.932378 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17a151c-b01b-4ba3-a41d-66cbc4fefb87 service placement] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:41.955171 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17a151c-b01b-4ba3-a41d-66cbc4fefb87 service placement] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:41.968093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae30f2e-76cf-40c3-a7e5-df56af719fe6 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '176da4679fcb46148da9289fccdad5c2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:41.968093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae30f2e-76cf-40c3-a7e5-df56af719fe6 service placement] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:41.968093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae30f2e-76cf-40c3-a7e5-df56af719fe6 service placement] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:41.968093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae30f2e-76cf-40c3-a7e5-df56af719fe6 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:41.968487 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-6ae30f2e-76cf-40c3-a7e5-df56af719fe6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:41.981813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae30f2e-76cf-40c3-a7e5-df56af719fe6 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:41.983472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae30f2e-76cf-40c3-a7e5-df56af719fe6 service placement] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:41.994033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1187/2374] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:41 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:55:42.078026 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656f111c-7835-4fe8-9a25-6da91474f591 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:42.079551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-656f111c-7835-4fe8-9a25-6da91474f591 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:42.148339 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656f111c-7835-4fe8-9a25-6da91474f591 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:42.153072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9287bd-4322-4042-a9b9-3564d646a76a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:42.153708 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9287bd-4322-4042-a9b9-3564d646a76a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:42.153990 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9287bd-4322-4042-a9b9-3564d646a76a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:42.154289 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9287bd-4322-4042-a9b9-3564d646a76a admin admin] PATH_INFO: `/v3/users/ede7a69d69084b3583433547473532be` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:42.155455 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9287bd-4322-4042-a9b9-3564d646a76a admin admin] RBAC: Authorizing `identity:delete_user(user_id=ede7a69d69084b3583433547473532be)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:42.157382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9287bd-4322-4042-a9b9-3564d646a76a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:42.440942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4d9287bd-4322-4042-a9b9-3564d646a76a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ede7a69d69084b3583433547473532be'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:42.453962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4d9287bd-4322-4042-a9b9-3564d646a76a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ede7a69d69084b3583433547473532be'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:42.550872 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4d9287bd-4322-4042-a9b9-3564d646a76a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ede7a69d69084b3583433547473532be'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:42.552121 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4d9287bd-4322-4042-a9b9-3564d646a76a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ede7a69d69084b3583433547473532be'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:42.559964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4d9287bd-4322-4042-a9b9-3564d646a76a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ede7a69d69084b3583433547473532be'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:42.566717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1188/2375] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:55:42 2022] DELETE /identity/v3/users/ede7a69d69084b3583433547473532be => generated 0 bytes in 489 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:42.822259 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae30f2e-76cf-40c3-a7e5-df56af719fe6 service placement] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:42.844455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae30f2e-76cf-40c3-a7e5-df56af719fe6 service placement] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:42.903269 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60004d5a-76ca-4660-88e7-4cf0b6c6f9e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:42.909293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60004d5a-76ca-4660-88e7-4cf0b6c6f9e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:42.909673 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60004d5a-76ca-4660-88e7-4cf0b6c6f9e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:42.910394 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60004d5a-76ca-4660-88e7-4cf0b6c6f9e7 admin admin] PATH_INFO: `/v3/projects/37a1aef0396c463d8a10f0ee811e1f2a` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:42.913744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60004d5a-76ca-4660-88e7-4cf0b6c6f9e7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=37a1aef0396c463d8a10f0ee811e1f2a)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:42.916164 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60004d5a-76ca-4660-88e7-4cf0b6c6f9e7 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:43.052935 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-60004d5a-76ca-4660-88e7-4cf0b6c6f9e7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '37a1aef0396c463d8a10f0ee811e1f2a', 'request_id': 'req-60004d5a-76ca-4660-88e7-4cf0b6c6f9e7'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:43.122479 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-60004d5a-76ca-4660-88e7-4cf0b6c6f9e7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '37a1aef0396c463d8a10f0ee811e1f2a', 'request_id': 'req-60004d5a-76ca-4660-88e7-4cf0b6c6f9e7'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:43.128803 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-60004d5a-76ca-4660-88e7-4cf0b6c6f9e7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '37a1aef0396c463d8a10f0ee811e1f2a', 'request_id': 'req-60004d5a-76ca-4660-88e7-4cf0b6c6f9e7'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:43.140223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-60004d5a-76ca-4660-88e7-4cf0b6c6f9e7 admin admin] The token cache is being invalidate because project 37a1aef0396c463d8a10f0ee811e1f2a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:55:43.140553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-60004d5a-76ca-4660-88e7-4cf0b6c6f9e7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:43.145654 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-60004d5a-76ca-4660-88e7-4cf0b6c6f9e7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:43.145654 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1188/2376] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:55:42 2022] DELETE /identity/v3/projects/37a1aef0396c463d8a10f0ee811e1f2a => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:43.147235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9287bd-4322-4042-a9b9-3564d646a76a admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:43.148642 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4d9287bd-4322-4042-a9b9-3564d646a76a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:43.162412 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9287bd-4322-4042-a9b9-3564d646a76a admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:43.207152 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:43.220208 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8ac4a6-88da-4ec6-9749-a86907ac29d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:43.220850 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8ac4a6-88da-4ec6-9749-a86907ac29d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:43.221158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8ac4a6-88da-4ec6-9749-a86907ac29d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:43.221436 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8ac4a6-88da-4ec6-9749-a86907ac29d1 admin admin] PATH_INFO: `/v3/users/8b849355226c45208f80839623d3db1e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:43.222562 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8ac4a6-88da-4ec6-9749-a86907ac29d1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b849355226c45208f80839623d3db1e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:43.225270 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8ac4a6-88da-4ec6-9749-a86907ac29d1 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:43.562128 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7b8ac4a6-88da-4ec6-9749-a86907ac29d1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b849355226c45208f80839623d3db1e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:43.577536 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7b8ac4a6-88da-4ec6-9749-a86907ac29d1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b849355226c45208f80839623d3db1e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:43.699715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7b8ac4a6-88da-4ec6-9749-a86907ac29d1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b849355226c45208f80839623d3db1e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:43.700617 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7b8ac4a6-88da-4ec6-9749-a86907ac29d1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b849355226c45208f80839623d3db1e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:43.710763 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7b8ac4a6-88da-4ec6-9749-a86907ac29d1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b849355226c45208f80839623d3db1e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:43.717981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1189/2377] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:55:43 2022] DELETE /identity/v3/users/8b849355226c45208f80839623d3db1e => generated 0 bytes in 571 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:44.049954 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60004d5a-76ca-4660-88e7-4cf0b6c6f9e7 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:44.052291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-60004d5a-76ca-4660-88e7-4cf0b6c6f9e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:44.062520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:44.072623 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60004d5a-76ca-4660-88e7-4cf0b6c6f9e7 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:44.149723 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1669416f-316d-422c-83dc-ec09c28f2fc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:44.150367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1669416f-316d-422c-83dc-ec09c28f2fc0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:44.150657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1669416f-316d-422c-83dc-ec09c28f2fc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:44.150959 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1669416f-316d-422c-83dc-ec09c28f2fc0 admin admin] PATH_INFO: `/v3/projects/80a5db390a074d4493fd3b17297ac181` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:44.152225 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1669416f-316d-422c-83dc-ec09c28f2fc0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=80a5db390a074d4493fd3b17297ac181)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:44.154198 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1669416f-316d-422c-83dc-ec09c28f2fc0 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:44.296530 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1669416f-316d-422c-83dc-ec09c28f2fc0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '80a5db390a074d4493fd3b17297ac181', 'request_id': 'req-1669416f-316d-422c-83dc-ec09c28f2fc0'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:44.407858 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1669416f-316d-422c-83dc-ec09c28f2fc0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '80a5db390a074d4493fd3b17297ac181', 'request_id': 'req-1669416f-316d-422c-83dc-ec09c28f2fc0'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:44.414420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1669416f-316d-422c-83dc-ec09c28f2fc0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '80a5db390a074d4493fd3b17297ac181', 'request_id': 'req-1669416f-316d-422c-83dc-ec09c28f2fc0'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:44.426161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1669416f-316d-422c-83dc-ec09c28f2fc0 admin admin] The token cache is being invalidate because project 80a5db390a074d4493fd3b17297ac181 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:55:44.427193 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1669416f-316d-422c-83dc-ec09c28f2fc0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:44.429853 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1669416f-316d-422c-83dc-ec09c28f2fc0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:44.429853 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1189/2378] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:55:44 2022] DELETE /identity/v3/projects/80a5db390a074d4493fd3b17297ac181 => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:46.155074 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8ac4a6-88da-4ec6-9749-a86907ac29d1 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:46.156493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7b8ac4a6-88da-4ec6-9749-a86907ac29d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:46.217044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8ac4a6-88da-4ec6-9749-a86907ac29d1 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:46.268046 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d9f522-a8e8-45e3-bd2d-59a2310f3bc9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:46.268656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d9f522-a8e8-45e3-bd2d-59a2310f3bc9 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:46.268935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d9f522-a8e8-45e3-bd2d-59a2310f3bc9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:46.269208 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d9f522-a8e8-45e3-bd2d-59a2310f3bc9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:46.271581 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b2d9f522-a8e8-45e3-bd2d-59a2310f3bc9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:46.314513 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:46.346413 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d9f522-a8e8-45e3-bd2d-59a2310f3bc9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:46.347893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d9f522-a8e8-45e3-bd2d-59a2310f3bc9 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:46.361924 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1190/2379] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:55:47.600045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1669416f-316d-422c-83dc-ec09c28f2fc0 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:47.601666 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1669416f-316d-422c-83dc-ec09c28f2fc0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:47.615984 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1669416f-316d-422c-83dc-ec09c28f2fc0 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:47.620743 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021ece9a-93f9-4e46-b7cc-cfca2a43b50f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:47.621418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021ece9a-93f9-4e46-b7cc-cfca2a43b50f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:47.621718 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021ece9a-93f9-4e46-b7cc-cfca2a43b50f admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:47.621987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021ece9a-93f9-4e46-b7cc-cfca2a43b50f admin admin] PATH_INFO: `/v3/users/939b3bedc9b34751899c7cee609168ef` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:47.623788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021ece9a-93f9-4e46-b7cc-cfca2a43b50f admin admin] RBAC: Authorizing `identity:delete_user(user_id=939b3bedc9b34751899c7cee609168ef)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:47.625901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021ece9a-93f9-4e46-b7cc-cfca2a43b50f admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:47.903899 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-021ece9a-93f9-4e46-b7cc-cfca2a43b50f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '939b3bedc9b34751899c7cee609168ef'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:48.008295 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-021ece9a-93f9-4e46-b7cc-cfca2a43b50f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '939b3bedc9b34751899c7cee609168ef'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:48.028718 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-021ece9a-93f9-4e46-b7cc-cfca2a43b50f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '939b3bedc9b34751899c7cee609168ef'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:48.029653 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-021ece9a-93f9-4e46-b7cc-cfca2a43b50f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '939b3bedc9b34751899c7cee609168ef'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:48.038900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-021ece9a-93f9-4e46-b7cc-cfca2a43b50f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '939b3bedc9b34751899c7cee609168ef'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:48.047764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1190/2380] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:55:47 2022] DELETE /identity/v3/users/939b3bedc9b34751899c7cee609168ef => generated 0 bytes in 448 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:48.052842 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6b2545-6495-4a34-b50e-db0ba78aed2d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:48.054572 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6b2545-6495-4a34-b50e-db0ba78aed2d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:48.055107 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6b2545-6495-4a34-b50e-db0ba78aed2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:48.077165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ef6b2545-6495-4a34-b50e-db0ba78aed2d None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:48.078489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ef6b2545-6495-4a34-b50e-db0ba78aed2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:48.151039 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1191/2381] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:55:48.162578 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021ece9a-93f9-4e46-b7cc-cfca2a43b50f admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:48.164178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-021ece9a-93f9-4e46-b7cc-cfca2a43b50f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:48.170857 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:48.222615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021ece9a-93f9-4e46-b7cc-cfca2a43b50f admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:48.228092 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5471d8d3-a8f7-4762-8487-79718f4dca72 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:48.228734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5471d8d3-a8f7-4762-8487-79718f4dca72 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:48.229047 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5471d8d3-a8f7-4762-8487-79718f4dca72 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:48.229400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5471d8d3-a8f7-4762-8487-79718f4dca72 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:48.240530 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5471d8d3-a8f7-4762-8487-79718f4dca72 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:48.242130 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5471d8d3-a8f7-4762-8487-79718f4dca72 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:48.264935 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1191/2382] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:55:48.839732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6b2545-6495-4a34-b50e-db0ba78aed2d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:48.841294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ef6b2545-6495-4a34-b50e-db0ba78aed2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:48.855963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6b2545-6495-4a34-b50e-db0ba78aed2d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:48.861171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10596e4f-a440-45ef-89a9-d6b14d608cdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:48.861937 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10596e4f-a440-45ef-89a9-d6b14d608cdc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:48.862228 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10596e4f-a440-45ef-89a9-d6b14d608cdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:48.862505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10596e4f-a440-45ef-89a9-d6b14d608cdc admin admin] PATH_INFO: `/v3/projects/1da1b7daef124a18bba699cd70b10e62` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:48.863742 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10596e4f-a440-45ef-89a9-d6b14d608cdc admin admin] RBAC: Authorizing `identity:delete_project(project_id=1da1b7daef124a18bba699cd70b10e62)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:48.865792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10596e4f-a440-45ef-89a9-d6b14d608cdc admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:49.027694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-10596e4f-a440-45ef-89a9-d6b14d608cdc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1da1b7daef124a18bba699cd70b10e62', 'request_id': 'req-10596e4f-a440-45ef-89a9-d6b14d608cdc'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:49.033928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-10596e4f-a440-45ef-89a9-d6b14d608cdc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1da1b7daef124a18bba699cd70b10e62', 'request_id': 'req-10596e4f-a440-45ef-89a9-d6b14d608cdc'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:49.116549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-10596e4f-a440-45ef-89a9-d6b14d608cdc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1da1b7daef124a18bba699cd70b10e62', 'request_id': 'req-10596e4f-a440-45ef-89a9-d6b14d608cdc'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:49.139383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-10596e4f-a440-45ef-89a9-d6b14d608cdc admin admin] The token cache is being invalidate because project 1da1b7daef124a18bba699cd70b10e62 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:55:49.139892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-10596e4f-a440-45ef-89a9-d6b14d608cdc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:49.140608 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-10596e4f-a440-45ef-89a9-d6b14d608cdc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:49.144290 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1192/2383] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:55:48 2022] DELETE /identity/v3/projects/1da1b7daef124a18bba699cd70b10e62 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:49.149421 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5471d8d3-a8f7-4762-8487-79718f4dca72 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:49.164264 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5471d8d3-a8f7-4762-8487-79718f4dca72 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:49.261979 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a215279-d928-4f86-b1b0-d70ee5a46e42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:49.274166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a215279-d928-4f86-b1b0-d70ee5a46e42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:49.274543 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a215279-d928-4f86-b1b0-d70ee5a46e42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:49.279148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a215279-d928-4f86-b1b0-d70ee5a46e42 admin admin] PATH_INFO: `/v3/users/0a982d07f9b04649b841c271a4f14174` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:49.281721 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a215279-d928-4f86-b1b0-d70ee5a46e42 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0a982d07f9b04649b841c271a4f14174)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:49.283811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a215279-d928-4f86-b1b0-d70ee5a46e42 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:49.510601 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-9a215279-d928-4f86-b1b0-d70ee5a46e42 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0a982d07f9b04649b841c271a4f14174'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:49.636818 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-9a215279-d928-4f86-b1b0-d70ee5a46e42 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0a982d07f9b04649b841c271a4f14174'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:49.656397 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-9a215279-d928-4f86-b1b0-d70ee5a46e42 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0a982d07f9b04649b841c271a4f14174'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:49.657135 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-9a215279-d928-4f86-b1b0-d70ee5a46e42 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0a982d07f9b04649b841c271a4f14174'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:49.665897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-9a215279-d928-4f86-b1b0-d70ee5a46e42 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0a982d07f9b04649b841c271a4f14174'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:49.675178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1192/2384] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:55:49 2022] DELETE /identity/v3/users/0a982d07f9b04649b841c271a4f14174 => generated 0 bytes in 525 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:50.375450 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10596e4f-a440-45ef-89a9-d6b14d608cdc admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:50.375450 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-10596e4f-a440-45ef-89a9-d6b14d608cdc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:50.409933 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10596e4f-a440-45ef-89a9-d6b14d608cdc admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:50.477495 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:50.487889 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e14c18-f050-402e-bf89-707db896890b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:50.489526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e14c18-f050-402e-bf89-707db896890b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:50.490184 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e14c18-f050-402e-bf89-707db896890b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:50.490677 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e14c18-f050-402e-bf89-707db896890b admin admin] PATH_INFO: `/v3/projects/fd04ba91ce40452abed64ba2516e3333` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:50.512365 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e14c18-f050-402e-bf89-707db896890b admin admin] RBAC: Authorizing `identity:delete_project(project_id=fd04ba91ce40452abed64ba2516e3333)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:50.518569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e14c18-f050-402e-bf89-707db896890b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:50.741911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a5e14c18-f050-402e-bf89-707db896890b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fd04ba91ce40452abed64ba2516e3333', 'request_id': 'req-a5e14c18-f050-402e-bf89-707db896890b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:50.751244 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a5e14c18-f050-402e-bf89-707db896890b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fd04ba91ce40452abed64ba2516e3333', 'request_id': 'req-a5e14c18-f050-402e-bf89-707db896890b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:50.888125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a5e14c18-f050-402e-bf89-707db896890b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fd04ba91ce40452abed64ba2516e3333', 'request_id': 'req-a5e14c18-f050-402e-bf89-707db896890b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:50.904726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a5e14c18-f050-402e-bf89-707db896890b admin admin] The token cache is being invalidate because project fd04ba91ce40452abed64ba2516e3333 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:55:50.905008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a5e14c18-f050-402e-bf89-707db896890b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:50.905761 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a5e14c18-f050-402e-bf89-707db896890b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:50.907401 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1193/2385] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:55:50 2022] DELETE /identity/v3/projects/fd04ba91ce40452abed64ba2516e3333 => generated 0 bytes in 538 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:50.914343 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfa38d6-3ab1-414e-9f49-570f86deb68b None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:50.914850 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfa38d6-3ab1-414e-9f49-570f86deb68b None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:50.915074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfa38d6-3ab1-414e-9f49-570f86deb68b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:50.946677 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-3dfa38d6-3ab1-414e-9f49-570f86deb68b None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:50.948227 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3dfa38d6-3ab1-414e-9f49-570f86deb68b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:50.982078 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:51.028772 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1193/2386] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:55:51.037976 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a918fb-dd12-4cf8-9202-030728e299b4 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:51.038270 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a918fb-dd12-4cf8-9202-030728e299b4 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:51.038552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a918fb-dd12-4cf8-9202-030728e299b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:51.064898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-85a918fb-dd12-4cf8-9202-030728e299b4 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:51.069084 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-85a918fb-dd12-4cf8-9202-030728e299b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:51.084945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1194/2387] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:55:51.088895 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfa38d6-3ab1-414e-9f49-570f86deb68b None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:51.101104 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfa38d6-3ab1-414e-9f49-570f86deb68b None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:51.111267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f39f8f-738a-4032-a861-961aeb036355 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:51.112452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f39f8f-738a-4032-a861-961aeb036355 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:51.113046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f39f8f-738a-4032-a861-961aeb036355 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:51.113338 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f39f8f-738a-4032-a861-961aeb036355 admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:51.114654 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f39f8f-738a-4032-a861-961aeb036355 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:55:51.115190 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f39f8f-738a-4032-a861-961aeb036355 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:51.116987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f39f8f-738a-4032-a861-961aeb036355 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:51.130787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1194/2388] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:55:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:55:51.146383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a918fb-dd12-4cf8-9202-030728e299b4 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:51.156895 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a918fb-dd12-4cf8-9202-030728e299b4 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:51.163958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ba16ce-19d3-4bcf-8b2f-427be929d124 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:51.164907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ba16ce-19d3-4bcf-8b2f-427be929d124 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:51.165469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ba16ce-19d3-4bcf-8b2f-427be929d124 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:51.165935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ba16ce-19d3-4bcf-8b2f-427be929d124 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:51.166784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ba16ce-19d3-4bcf-8b2f-427be929d124 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:51.169112 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ba16ce-19d3-4bcf-8b2f-427be929d124 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:51.346013 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1195/2389] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:55:51 2022] POST /identity/v3/projects => generated 361 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:55:51.350645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f39f8f-738a-4032-a861-961aeb036355 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:51.361489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f39f8f-738a-4032-a861-961aeb036355 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:51.422731 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3b020c-d78e-4dc0-a53b-195b2e032cc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:51.423374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3b020c-d78e-4dc0-a53b-195b2e032cc8 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:51.423662 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3b020c-d78e-4dc0-a53b-195b2e032cc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:51.423989 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3b020c-d78e-4dc0-a53b-195b2e032cc8 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:51.424632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3b020c-d78e-4dc0-a53b-195b2e032cc8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:51.426644 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3b020c-d78e-4dc0-a53b-195b2e032cc8 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:51.474244 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ba16ce-19d3-4bcf-8b2f-427be929d124 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:51.483895 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-38ba16ce-19d3-4bcf-8b2f-427be929d124 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:51.499214 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ba16ce-19d3-4bcf-8b2f-427be929d124 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:51.504082 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8983d355-0c1f-41b5-a3d6-225234c5c00e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:51.504831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8983d355-0c1f-41b5-a3d6-225234c5c00e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:51.505157 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8983d355-0c1f-41b5-a3d6-225234c5c00e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:51.505756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8983d355-0c1f-41b5-a3d6-225234c5c00e admin admin] PATH_INFO: `/v3/users/8f410d691e364880b13bb9144bbc5e7d` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:51.506930 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8983d355-0c1f-41b5-a3d6-225234c5c00e admin admin] RBAC: Authorizing `identity:delete_user(user_id=8f410d691e364880b13bb9144bbc5e7d)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:51.508849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8983d355-0c1f-41b5-a3d6-225234c5c00e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:51.563535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1195/2390] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:55:51 2022] POST /identity/v3/users => generated 281 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:55:51.567485 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3b020c-d78e-4dc0-a53b-195b2e032cc8 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:51.576837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3b020c-d78e-4dc0-a53b-195b2e032cc8 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:51.581603 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439d22b1-a5a6-4ea1-9bf6-0511c5cd8d92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:51.582163 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439d22b1-a5a6-4ea1-9bf6-0511c5cd8d92 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:51.583117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439d22b1-a5a6-4ea1-9bf6-0511c5cd8d92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:51.583448 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439d22b1-a5a6-4ea1-9bf6-0511c5cd8d92 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:51.584069 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439d22b1-a5a6-4ea1-9bf6-0511c5cd8d92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:51.585643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439d22b1-a5a6-4ea1-9bf6-0511c5cd8d92 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:51.597335 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1196/2391] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:55:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:55:51.602088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439d22b1-a5a6-4ea1-9bf6-0511c5cd8d92 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:51.612615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439d22b1-a5a6-4ea1-9bf6-0511c5cd8d92 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:51.617740 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03a3b86-68ef-45ee-9b40-e711f1c5ba01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:51.618348 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03a3b86-68ef-45ee-9b40-e711f1c5ba01 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:51.618633 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03a3b86-68ef-45ee-9b40-e711f1c5ba01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:51.619078 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03a3b86-68ef-45ee-9b40-e711f1c5ba01 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:51.619576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03a3b86-68ef-45ee-9b40-e711f1c5ba01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:51.621195 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03a3b86-68ef-45ee-9b40-e711f1c5ba01 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:51.633823 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1197/2392] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:55:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:55:51.637716 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03a3b86-68ef-45ee-9b40-e711f1c5ba01 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:51.648491 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03a3b86-68ef-45ee-9b40-e711f1c5ba01 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:51.653147 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada57610-b92f-4645-b006-d02e41f97f51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:51.653912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada57610-b92f-4645-b006-d02e41f97f51 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:51.654084 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada57610-b92f-4645-b006-d02e41f97f51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:51.654359 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada57610-b92f-4645-b006-d02e41f97f51 admin admin] PATH_INFO: `/v3/projects/f9107c9a98024365a67e95ff1e6d685f/users/bfead81ef66d448485a4c464e983af61/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:51.669494 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada57610-b92f-4645-b006-d02e41f97f51 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f9107c9a98024365a67e95ff1e6d685f, user_id=bfead81ef66d448485a4c464e983af61, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:51.672180 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada57610-b92f-4645-b006-d02e41f97f51 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:51.831916 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1198/2393] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:55:51 2022] PUT /identity/v3/projects/f9107c9a98024365a67e95ff1e6d685f/users/bfead81ef66d448485a4c464e983af61/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:51.841264 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8983d355-0c1f-41b5-a3d6-225234c5c00e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8f410d691e364880b13bb9144bbc5e7d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:51.855794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8983d355-0c1f-41b5-a3d6-225234c5c00e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8f410d691e364880b13bb9144bbc5e7d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:51.903581 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d09bb79-858a-4248-bc47-85ba50614a89 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:51.903872 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d09bb79-858a-4248-bc47-85ba50614a89 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:51.904145 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d09bb79-858a-4248-bc47-85ba50614a89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:51.933261 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-3d09bb79-858a-4248-bc47-85ba50614a89 None None] MFA Rules not processed for user `bfead81ef66d448485a4c464e983af61`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:51.934523 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3d09bb79-858a-4248-bc47-85ba50614a89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:51.945410 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8983d355-0c1f-41b5-a3d6-225234c5c00e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8f410d691e364880b13bb9144bbc5e7d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:51.946177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8983d355-0c1f-41b5-a3d6-225234c5c00e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8f410d691e364880b13bb9144bbc5e7d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:51.954208 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8983d355-0c1f-41b5-a3d6-225234c5c00e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8f410d691e364880b13bb9144bbc5e7d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:51.960855 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1196/2394] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:55:51 2022] DELETE /identity/v3/users/8f410d691e364880b13bb9144bbc5e7d => generated 0 bytes in 488 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:55:52.011139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1199/2395] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:51 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:55:52.046187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8983d355-0c1f-41b5-a3d6-225234c5c00e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:52.048932 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8983d355-0c1f-41b5-a3d6-225234c5c00e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:52.116870 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8983d355-0c1f-41b5-a3d6-225234c5c00e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:52.120643 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdbefda-75ec-4835-af0f-807e2a8b59e8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:52.121262 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdbefda-75ec-4835-af0f-807e2a8b59e8 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:52.121548 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdbefda-75ec-4835-af0f-807e2a8b59e8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:52.121849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdbefda-75ec-4835-af0f-807e2a8b59e8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:52.200884 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdbefda-75ec-4835-af0f-807e2a8b59e8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:52.202898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdbefda-75ec-4835-af0f-807e2a8b59e8 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:52.212155 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1197/2396] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:52 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:55:52.228555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d09bb79-858a-4248-bc47-85ba50614a89 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:52.230959 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3d09bb79-858a-4248-bc47-85ba50614a89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:52.296835 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d09bb79-858a-4248-bc47-85ba50614a89 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:52.301418 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdbefda-75ec-4835-af0f-807e2a8b59e8 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:52.303781 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ffdbefda-75ec-4835-af0f-807e2a8b59e8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:52.384677 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb3ce55-1b18-4e8a-af72-117b031ca569 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:52.384677 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb3ce55-1b18-4e8a-af72-117b031ca569 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:52.384677 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb3ce55-1b18-4e8a-af72-117b031ca569 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:52.384677 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb3ce55-1b18-4e8a-af72-117b031ca569 admin admin] PATH_INFO: `/v3/users/bfead81ef66d448485a4c464e983af61` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:52.385167 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb3ce55-1b18-4e8a-af72-117b031ca569 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bfead81ef66d448485a4c464e983af61)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:52.386533 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb3ce55-1b18-4e8a-af72-117b031ca569 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:52.402933 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdbefda-75ec-4835-af0f-807e2a8b59e8 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:52.408950 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bdc5e8-0462-4f51-8d6e-71ce6ae1cefe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:52.409644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bdc5e8-0462-4f51-8d6e-71ce6ae1cefe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:52.414335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bdc5e8-0462-4f51-8d6e-71ce6ae1cefe admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:52.414335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bdc5e8-0462-4f51-8d6e-71ce6ae1cefe admin admin] PATH_INFO: `/v3/projects/e94b50976295422c9c8374155cf2d2c2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:52.414335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bdc5e8-0462-4f51-8d6e-71ce6ae1cefe admin admin] RBAC: Authorizing `identity:delete_project(project_id=e94b50976295422c9c8374155cf2d2c2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:52.414335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bdc5e8-0462-4f51-8d6e-71ce6ae1cefe admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:52.575019 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=80,keystone:DELETE=17,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:52.619203 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a6bdc5e8-0462-4f51-8d6e-71ce6ae1cefe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e94b50976295422c9c8374155cf2d2c2', 'request_id': 'req-a6bdc5e8-0462-4f51-8d6e-71ce6ae1cefe'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:52.619454 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1cb3ce55-1b18-4e8a-af72-117b031ca569 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bfead81ef66d448485a4c464e983af61'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:52.627194 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a6bdc5e8-0462-4f51-8d6e-71ce6ae1cefe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e94b50976295422c9c8374155cf2d2c2', 'request_id': 'req-a6bdc5e8-0462-4f51-8d6e-71ce6ae1cefe'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:52.705478 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1cb3ce55-1b18-4e8a-af72-117b031ca569 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bfead81ef66d448485a4c464e983af61'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:52.719032 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1cb3ce55-1b18-4e8a-af72-117b031ca569 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bfead81ef66d448485a4c464e983af61'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:52.719998 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1cb3ce55-1b18-4e8a-af72-117b031ca569 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bfead81ef66d448485a4c464e983af61'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:52.741813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1cb3ce55-1b18-4e8a-af72-117b031ca569 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bfead81ef66d448485a4c464e983af61'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:52.755028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1200/2397] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:55:52 2022] DELETE /identity/v3/users/bfead81ef66d448485a4c464e983af61 => generated 0 bytes in 523 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:52.757123 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d8c3c7-9c24-43cf-8713-26c6fbf98b67 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:52.757541 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d8c3c7-9c24-43cf-8713-26c6fbf98b67 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:52.757931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d8c3c7-9c24-43cf-8713-26c6fbf98b67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:52.763758 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a6bdc5e8-0462-4f51-8d6e-71ce6ae1cefe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e94b50976295422c9c8374155cf2d2c2', 'request_id': 'req-a6bdc5e8-0462-4f51-8d6e-71ce6ae1cefe'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:52.785212 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a6bdc5e8-0462-4f51-8d6e-71ce6ae1cefe admin admin] The token cache is being invalidate because project e94b50976295422c9c8374155cf2d2c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:55:52.785507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a6bdc5e8-0462-4f51-8d6e-71ce6ae1cefe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:52.786179 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a6bdc5e8-0462-4f51-8d6e-71ce6ae1cefe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:52.788200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-61d8c3c7-9c24-43cf-8713-26c6fbf98b67 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:52.789045 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1198/2398] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:55:52 2022] DELETE /identity/v3/projects/e94b50976295422c9c8374155cf2d2c2 => generated 0 bytes in 488 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:55:52.791419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-61d8c3c7-9c24-43cf-8713-26c6fbf98b67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:52.794601 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acf36c2-3bc2-4d6b-8ac9-986852cccfaf None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:52.796674 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acf36c2-3bc2-4d6b-8ac9-986852cccfaf None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:52.796674 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acf36c2-3bc2-4d6b-8ac9-986852cccfaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:52.816027 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8acf36c2-3bc2-4d6b-8ac9-986852cccfaf None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:52.817216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8acf36c2-3bc2-4d6b-8ac9-986852cccfaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:52.852328 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1201/2399] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:55:52.863894 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d8c3c7-9c24-43cf-8713-26c6fbf98b67 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:52.865629 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-61d8c3c7-9c24-43cf-8713-26c6fbf98b67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:52.880365 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1199/2400] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:55:52.889305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5333afa-c2c3-472c-8ab0-c61192dc945f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:52.892795 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5333afa-c2c3-472c-8ab0-c61192dc945f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:52.892795 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5333afa-c2c3-472c-8ab0-c61192dc945f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:52.910232 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f5333afa-c2c3-472c-8ab0-c61192dc945f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:52.911451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f5333afa-c2c3-472c-8ab0-c61192dc945f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:52.919763 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1200/2401] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:55:52.925453 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d8c3c7-9c24-43cf-8713-26c6fbf98b67 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:52.929942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5333afa-c2c3-472c-8ab0-c61192dc945f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:52.930081 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1ace5b-6e75-433a-9502-e0ecd794961e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:52.930731 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1ace5b-6e75-433a-9502-e0ecd794961e service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:52.931053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1ace5b-6e75-433a-9502-e0ecd794961e service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:52.931330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1ace5b-6e75-433a-9502-e0ecd794961e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:52.939699 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5333afa-c2c3-472c-8ab0-c61192dc945f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:52.942538 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1ace5b-6e75-433a-9502-e0ecd794961e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:52.945575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1ace5b-6e75-433a-9502-e0ecd794961e service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:52.946514 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b528d305-925e-4dbb-8832-80be57e9b003 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:52.947230 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b528d305-925e-4dbb-8832-80be57e9b003 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:52.947626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b528d305-925e-4dbb-8832-80be57e9b003 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:52.947967 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b528d305-925e-4dbb-8832-80be57e9b003 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:52.948716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b528d305-925e-4dbb-8832-80be57e9b003 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:55:52.949153 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b528d305-925e-4dbb-8832-80be57e9b003 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:52.950878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b528d305-925e-4dbb-8832-80be57e9b003 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:52.956764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1202/2402] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:55:52.967234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1201/2403] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:55:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:55:52.969284 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1ace5b-6e75-433a-9502-e0ecd794961e service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:52.980541 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1ace5b-6e75-433a-9502-e0ecd794961e service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:52.986691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de199b8b-b145-48ef-a1c6-ad460e08d617 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:52.987332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de199b8b-b145-48ef-a1c6-ad460e08d617 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:52.987612 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de199b8b-b145-48ef-a1c6-ad460e08d617 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:52.987904 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de199b8b-b145-48ef-a1c6-ad460e08d617 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:52.988525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de199b8b-b145-48ef-a1c6-ad460e08d617 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:52.991859 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de199b8b-b145-48ef-a1c6-ad460e08d617 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:53.101559 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1203/2404] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:55:52 2022] POST /identity/v3/projects => generated 367 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:55:53.105830 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b528d305-925e-4dbb-8832-80be57e9b003 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:53.116398 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b528d305-925e-4dbb-8832-80be57e9b003 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:53.175583 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8854a9a9-7038-4b0b-afcc-1682126c1e61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:53.176167 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8854a9a9-7038-4b0b-afcc-1682126c1e61 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:53.176446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8854a9a9-7038-4b0b-afcc-1682126c1e61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:53.176721 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8854a9a9-7038-4b0b-afcc-1682126c1e61 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:53.177389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8854a9a9-7038-4b0b-afcc-1682126c1e61 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:53.180029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8854a9a9-7038-4b0b-afcc-1682126c1e61 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:53.219835 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:53.334943 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1202/2405] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:55:53 2022] POST /identity/v3/users => generated 284 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:55:53.337739 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de199b8b-b145-48ef-a1c6-ad460e08d617 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:53.348327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de199b8b-b145-48ef-a1c6-ad460e08d617 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:53.353804 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e97ab4-e83e-4aed-be27-4ff58dac9474 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:53.354407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e97ab4-e83e-4aed-be27-4ff58dac9474 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:53.354688 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e97ab4-e83e-4aed-be27-4ff58dac9474 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:53.355126 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e97ab4-e83e-4aed-be27-4ff58dac9474 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:53.355834 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e97ab4-e83e-4aed-be27-4ff58dac9474 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:53.357532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e97ab4-e83e-4aed-be27-4ff58dac9474 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:53.368847 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1204/2406] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:55:53 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:55:53.373413 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8854a9a9-7038-4b0b-afcc-1682126c1e61 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:53.388088 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8854a9a9-7038-4b0b-afcc-1682126c1e61 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:53.396977 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5f0006-554b-4fd4-82aa-12cd1bd056ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:53.397603 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5f0006-554b-4fd4-82aa-12cd1bd056ee admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:53.397883 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5f0006-554b-4fd4-82aa-12cd1bd056ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:53.398156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5f0006-554b-4fd4-82aa-12cd1bd056ee admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:53.401170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5f0006-554b-4fd4-82aa-12cd1bd056ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:53.401170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5f0006-554b-4fd4-82aa-12cd1bd056ee admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:53.412958 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1203/2407] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:55:53 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:55:53.417553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e97ab4-e83e-4aed-be27-4ff58dac9474 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:53.427721 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e97ab4-e83e-4aed-be27-4ff58dac9474 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:53.437516 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acc3130-d675-4730-a052-508a5d6bf0b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:53.437516 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acc3130-d675-4730-a052-508a5d6bf0b2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:53.437516 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acc3130-d675-4730-a052-508a5d6bf0b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:53.437516 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acc3130-d675-4730-a052-508a5d6bf0b2 admin admin] PATH_INFO: `/v3/projects/a698a207b126483ab7659f5dc9e280e8/users/e7467dbdeede4c2ebafe9a255d48939d/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:53.451282 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acc3130-d675-4730-a052-508a5d6bf0b2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a698a207b126483ab7659f5dc9e280e8, user_id=e7467dbdeede4c2ebafe9a255d48939d, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:53.454207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acc3130-d675-4730-a052-508a5d6bf0b2 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:53.455733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5f0006-554b-4fd4-82aa-12cd1bd056ee admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:53.458687 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2a5f0006-554b-4fd4-82aa-12cd1bd056ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:53.473051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5f0006-554b-4fd4-82aa-12cd1bd056ee admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:53.479459 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0013dd90-ef32-454c-935e-a5b025cbac4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:53.480243 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0013dd90-ef32-454c-935e-a5b025cbac4f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:53.480526 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0013dd90-ef32-454c-935e-a5b025cbac4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:53.481296 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0013dd90-ef32-454c-935e-a5b025cbac4f admin admin] PATH_INFO: `/v3/projects/f9107c9a98024365a67e95ff1e6d685f` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:53.482054 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0013dd90-ef32-454c-935e-a5b025cbac4f admin admin] RBAC: Authorizing `identity:delete_project(project_id=f9107c9a98024365a67e95ff1e6d685f)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:53.484012 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0013dd90-ef32-454c-935e-a5b025cbac4f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:53.529958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1205/2408] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:55:53 2022] PUT /identity/v3/projects/a698a207b126483ab7659f5dc9e280e8/users/e7467dbdeede4c2ebafe9a255d48939d/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:53.546237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9958877e-58cd-4277-aeec-ad8fb55bf762 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:53.546525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9958877e-58cd-4277-aeec-ad8fb55bf762 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:53.546842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9958877e-58cd-4277-aeec-ad8fb55bf762 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:53.575432 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-9958877e-58cd-4277-aeec-ad8fb55bf762 None None] MFA Rules not processed for user `e7467dbdeede4c2ebafe9a255d48939d`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:53.576640 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-9958877e-58cd-4277-aeec-ad8fb55bf762 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:53.636258 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0013dd90-ef32-454c-935e-a5b025cbac4f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f9107c9a98024365a67e95ff1e6d685f', 'request_id': 'req-0013dd90-ef32-454c-935e-a5b025cbac4f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:53.643388 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1206/2409] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:53 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:55:53.643551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0013dd90-ef32-454c-935e-a5b025cbac4f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f9107c9a98024365a67e95ff1e6d685f', 'request_id': 'req-0013dd90-ef32-454c-935e-a5b025cbac4f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:53.654868 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9958877e-58cd-4277-aeec-ad8fb55bf762 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:53.656252 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-9958877e-58cd-4277-aeec-ad8fb55bf762 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:53.712848 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9958877e-58cd-4277-aeec-ad8fb55bf762 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:53.763749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0013dd90-ef32-454c-935e-a5b025cbac4f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f9107c9a98024365a67e95ff1e6d685f', 'request_id': 'req-0013dd90-ef32-454c-935e-a5b025cbac4f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:53.772748 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2035c8-0139-46e6-80c1-d84838d2c839 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:53.773387 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2035c8-0139-46e6-80c1-d84838d2c839 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:53.773663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2035c8-0139-46e6-80c1-d84838d2c839 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:53.773936 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2035c8-0139-46e6-80c1-d84838d2c839 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:53.777957 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0013dd90-ef32-454c-935e-a5b025cbac4f admin admin] The token cache is being invalidate because project f9107c9a98024365a67e95ff1e6d685f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:55:53.778279 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0013dd90-ef32-454c-935e-a5b025cbac4f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:53.779097 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0013dd90-ef32-454c-935e-a5b025cbac4f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:53.780764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1204/2410] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:55:53 2022] DELETE /identity/v3/projects/f9107c9a98024365a67e95ff1e6d685f => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:55:53.787643 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0d3fc6-3970-44d6-9870-e8f31019268f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:53.787643 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0d3fc6-3970-44d6-9870-e8f31019268f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:53.787643 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0d3fc6-3970-44d6-9870-e8f31019268f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:53.808360 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9f0d3fc6-3970-44d6-9870-e8f31019268f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:53.810539 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9f0d3fc6-3970-44d6-9870-e8f31019268f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:53.839547 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2035c8-0139-46e6-80c1-d84838d2c839 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:53.841070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2035c8-0139-46e6-80c1-d84838d2c839 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:53.850452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1207/2411] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:53 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:55:53.873580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1205/2412] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:55:53.881828 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465c32b8-1515-4d80-ab80-eace367a81f6 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:53.887301 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465c32b8-1515-4d80-ab80-eace367a81f6 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:53.887301 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465c32b8-1515-4d80-ab80-eace367a81f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:53.903446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-465c32b8-1515-4d80-ab80-eace367a81f6 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:53.905066 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-465c32b8-1515-4d80-ab80-eace367a81f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:53.913628 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1208/2413] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:55:53.918031 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0d3fc6-3970-44d6-9870-e8f31019268f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:53.927827 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0d3fc6-3970-44d6-9870-e8f31019268f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:53.934108 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d0461c-0a61-4ae1-819a-94df282ded45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:53.934721 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d0461c-0a61-4ae1-819a-94df282ded45 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:53.935032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d0461c-0a61-4ae1-819a-94df282ded45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:53.935308 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d0461c-0a61-4ae1-819a-94df282ded45 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:53.935954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d0461c-0a61-4ae1-819a-94df282ded45 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:55:53.936257 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d0461c-0a61-4ae1-819a-94df282ded45 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:53.937842 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d0461c-0a61-4ae1-819a-94df282ded45 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:53.950525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1206/2414] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:55:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:55:53.955121 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465c32b8-1515-4d80-ab80-eace367a81f6 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:53.965479 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465c32b8-1515-4d80-ab80-eace367a81f6 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:53.969634 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2970a02f-b52e-4621-ad23-e5de00086d13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:53.970225 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2970a02f-b52e-4621-ad23-e5de00086d13 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:53.970502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2970a02f-b52e-4621-ad23-e5de00086d13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:53.970772 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2970a02f-b52e-4621-ad23-e5de00086d13 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:53.971432 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2970a02f-b52e-4621-ad23-e5de00086d13 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:53.973338 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2970a02f-b52e-4621-ad23-e5de00086d13 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:54.119763 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1209/2415] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:55:53 2022] POST /identity/v3/projects => generated 375 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:55:54.123550 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d0461c-0a61-4ae1-819a-94df282ded45 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:54.135564 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d0461c-0a61-4ae1-819a-94df282ded45 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:54.194529 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ac9489-cf61-49c2-aa37-d18267a8e040 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:54.195168 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ac9489-cf61-49c2-aa37-d18267a8e040 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:54.195446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ac9489-cf61-49c2-aa37-d18267a8e040 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:54.195717 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ac9489-cf61-49c2-aa37-d18267a8e040 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:54.196331 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ac9489-cf61-49c2-aa37-d18267a8e040 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:54.198286 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ac9489-cf61-49c2-aa37-d18267a8e040 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:54.230939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:54.306556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1207/2416] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:55:54 2022] POST /identity/v3/users => generated 288 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:55:54.310326 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2970a02f-b52e-4621-ad23-e5de00086d13 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:54.320763 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2970a02f-b52e-4621-ad23-e5de00086d13 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:54.329152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59cda2c-36db-4bb1-828e-25a4c824b841 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:54.329670 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59cda2c-36db-4bb1-828e-25a4c824b841 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:54.329895 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59cda2c-36db-4bb1-828e-25a4c824b841 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:54.330174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59cda2c-36db-4bb1-828e-25a4c824b841 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:54.331388 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59cda2c-36db-4bb1-828e-25a4c824b841 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:54.332438 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59cda2c-36db-4bb1-828e-25a4c824b841 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:54.350435 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1210/2417] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:55:54 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:55:54.351845 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ac9489-cf61-49c2-aa37-d18267a8e040 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:54.362171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ac9489-cf61-49c2-aa37-d18267a8e040 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:54.367362 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71dc48c5-bf93-44ec-a866-da4ca3effdca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:54.367988 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71dc48c5-bf93-44ec-a866-da4ca3effdca admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:54.368267 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71dc48c5-bf93-44ec-a866-da4ca3effdca admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:54.369010 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71dc48c5-bf93-44ec-a866-da4ca3effdca admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:54.369654 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71dc48c5-bf93-44ec-a866-da4ca3effdca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:54.371262 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71dc48c5-bf93-44ec-a866-da4ca3effdca admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:54.382532 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1208/2418] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:55:54 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:55:54.391582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59cda2c-36db-4bb1-828e-25a4c824b841 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:54.402837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59cda2c-36db-4bb1-828e-25a4c824b841 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:54.407772 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febde98c-2b2c-4395-9635-ff6fedf51f02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:54.408621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febde98c-2b2c-4395-9635-ff6fedf51f02 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:54.409983 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:54.410430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febde98c-2b2c-4395-9635-ff6fedf51f02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:54.414580 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febde98c-2b2c-4395-9635-ff6fedf51f02 admin admin] PATH_INFO: `/v3/projects/d45b537510f8486f8199e4b51f31e143/users/1bc9d5fb65bc4014b9673c441a4fdd5d/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:54.427640 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febde98c-2b2c-4395-9635-ff6fedf51f02 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d45b537510f8486f8199e4b51f31e143, user_id=1bc9d5fb65bc4014b9673c441a4fdd5d, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:54.430454 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febde98c-2b2c-4395-9635-ff6fedf51f02 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:54.505764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1211/2419] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:55:54 2022] PUT /identity/v3/projects/d45b537510f8486f8199e4b51f31e143/users/1bc9d5fb65bc4014b9673c441a4fdd5d/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:54.518853 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b3a65d-9356-4313-96f0-f09b6071a201 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:54.519179 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b3a65d-9356-4313-96f0-f09b6071a201 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:54.519458 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b3a65d-9356-4313-96f0-f09b6071a201 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:54.555464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-12b3a65d-9356-4313-96f0-f09b6071a201 None None] MFA Rules not processed for user `1bc9d5fb65bc4014b9673c441a4fdd5d`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:54.556766 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-12b3a65d-9356-4313-96f0-f09b6071a201 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:54.625596 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1209/2420] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:54 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:55:54.637523 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febde98c-2b2c-4395-9635-ff6fedf51f02 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:54.640028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-febde98c-2b2c-4395-9635-ff6fedf51f02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:54.697663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febde98c-2b2c-4395-9635-ff6fedf51f02 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:54.701630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d90caa-7a01-49f3-8ed9-9f0fe3841562 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:54.702236 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d90caa-7a01-49f3-8ed9-9f0fe3841562 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:54.702517 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d90caa-7a01-49f3-8ed9-9f0fe3841562 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:54.702792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d90caa-7a01-49f3-8ed9-9f0fe3841562 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:54.711850 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d90caa-7a01-49f3-8ed9-9f0fe3841562 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:54.713286 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d90caa-7a01-49f3-8ed9-9f0fe3841562 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:54.722363 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1212/2421] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:54 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:55:55.535193 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:55.574462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:55.996713 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:56.151130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:55:56.952994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b3a65d-9356-4313-96f0-f09b6071a201 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:56.954408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-12b3a65d-9356-4313-96f0-f09b6071a201 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:57.011352 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b3a65d-9356-4313-96f0-f09b6071a201 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:57.015218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e652d796-cc12-44be-914e-0a1d68245a93 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:57.017552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e652d796-cc12-44be-914e-0a1d68245a93 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:57.017552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e652d796-cc12-44be-914e-0a1d68245a93 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:57.017552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e652d796-cc12-44be-914e-0a1d68245a93 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:57.018030 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e652d796-cc12-44be-914e-0a1d68245a93 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:57.086468 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e652d796-cc12-44be-914e-0a1d68245a93 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:57.087939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e652d796-cc12-44be-914e-0a1d68245a93 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:57.097779 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1210/2422] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:55:59.058054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d90caa-7a01-49f3-8ed9-9f0fe3841562 service cinder] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:59.069160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d90caa-7a01-49f3-8ed9-9f0fe3841562 service cinder] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:59.073758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcefc77-bf3a-4c17-9896-eae431f94aba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:59.074377 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcefc77-bf3a-4c17-9896-eae431f94aba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:59.074675 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcefc77-bf3a-4c17-9896-eae431f94aba admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:59.075040 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcefc77-bf3a-4c17-9896-eae431f94aba admin admin] PATH_INFO: `/v3/users/1bc9d5fb65bc4014b9673c441a4fdd5d` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:59.076101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcefc77-bf3a-4c17-9896-eae431f94aba admin admin] RBAC: Authorizing `identity:delete_user(user_id=1bc9d5fb65bc4014b9673c441a4fdd5d)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:59.078050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcefc77-bf3a-4c17-9896-eae431f94aba admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:59.316663 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6fcefc77-bf3a-4c17-9896-eae431f94aba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1bc9d5fb65bc4014b9673c441a4fdd5d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:59.390444 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6fcefc77-bf3a-4c17-9896-eae431f94aba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1bc9d5fb65bc4014b9673c441a4fdd5d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:59.410894 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6fcefc77-bf3a-4c17-9896-eae431f94aba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1bc9d5fb65bc4014b9673c441a4fdd5d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:59.412212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6fcefc77-bf3a-4c17-9896-eae431f94aba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1bc9d5fb65bc4014b9673c441a4fdd5d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:59.422337 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6fcefc77-bf3a-4c17-9896-eae431f94aba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1bc9d5fb65bc4014b9673c441a4fdd5d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:55:59.435273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1213/2423] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:55:59 2022] DELETE /identity/v3/users/1bc9d5fb65bc4014b9673c441a4fdd5d => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:55:59.436290 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b0f40d-5b8d-4ac9-8711-426df5d108eb None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:59.436800 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b0f40d-5b8d-4ac9-8711-426df5d108eb None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:59.436901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b0f40d-5b8d-4ac9-8711-426df5d108eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:59.465550 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a7b0f40d-5b8d-4ac9-8711-426df5d108eb None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:55:59.465550 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a7b0f40d-5b8d-4ac9-8711-426df5d108eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:59.557087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1211/2424] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:55:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:55:59.564794 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcefc77-bf3a-4c17-9896-eae431f94aba admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:55:59.566539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-6fcefc77-bf3a-4c17-9896-eae431f94aba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:55:59.643525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcefc77-bf3a-4c17-9896-eae431f94aba admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:55:59.646924 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340d1eb4-1c94-49bc-8101-e2cc54f0263c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:55:59.647671 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340d1eb4-1c94-49bc-8101-e2cc54f0263c service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:55:59.648114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340d1eb4-1c94-49bc-8101-e2cc54f0263c service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:55:59.648463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340d1eb4-1c94-49bc-8101-e2cc54f0263c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:55:59.658358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340d1eb4-1c94-49bc-8101-e2cc54f0263c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:55:59.660008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340d1eb4-1c94-49bc-8101-e2cc54f0263c service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:55:59.671379 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1214/2425] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:55:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:00.206823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b0f40d-5b8d-4ac9-8711-426df5d108eb None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:00.208649 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a7b0f40d-5b8d-4ac9-8711-426df5d108eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:00.221714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b0f40d-5b8d-4ac9-8711-426df5d108eb None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:00.237483 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1635f1b6-f2c3-4eda-9010-63c749a32863 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:00.238158 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1635f1b6-f2c3-4eda-9010-63c749a32863 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:00.238441 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1635f1b6-f2c3-4eda-9010-63c749a32863 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:00.238723 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1635f1b6-f2c3-4eda-9010-63c749a32863 admin admin] PATH_INFO: `/v3/projects/d45b537510f8486f8199e4b51f31e143` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:00.241505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1635f1b6-f2c3-4eda-9010-63c749a32863 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d45b537510f8486f8199e4b51f31e143)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:00.247119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1635f1b6-f2c3-4eda-9010-63c749a32863 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:00.396435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:00.405149 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1635f1b6-f2c3-4eda-9010-63c749a32863 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd45b537510f8486f8199e4b51f31e143', 'request_id': 'req-1635f1b6-f2c3-4eda-9010-63c749a32863'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:00.411622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1635f1b6-f2c3-4eda-9010-63c749a32863 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd45b537510f8486f8199e4b51f31e143', 'request_id': 'req-1635f1b6-f2c3-4eda-9010-63c749a32863'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:00.477531 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1635f1b6-f2c3-4eda-9010-63c749a32863 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd45b537510f8486f8199e4b51f31e143', 'request_id': 'req-1635f1b6-f2c3-4eda-9010-63c749a32863'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:00.488480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1635f1b6-f2c3-4eda-9010-63c749a32863 admin admin] The token cache is being invalidate because project d45b537510f8486f8199e4b51f31e143 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:00.488775 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1635f1b6-f2c3-4eda-9010-63c749a32863 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:00.489396 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1635f1b6-f2c3-4eda-9010-63c749a32863 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:00.490882 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1212/2426] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:00 2022] DELETE /identity/v3/projects/d45b537510f8486f8199e4b51f31e143 => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:00.496739 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131c81d0-282d-4ca9-9aa0-bc294f2801fe None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:00.497206 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131c81d0-282d-4ca9-9aa0-bc294f2801fe None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:00.497314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131c81d0-282d-4ca9-9aa0-bc294f2801fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:00.529765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-131c81d0-282d-4ca9-9aa0-bc294f2801fe None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:00.531121 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-131c81d0-282d-4ca9-9aa0-bc294f2801fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:00.547282 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:00.617952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1215/2427] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:00.627584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c14a834-798b-4513-9225-75271242c67c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:00.627879 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c14a834-798b-4513-9225-75271242c67c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:00.628166 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c14a834-798b-4513-9225-75271242c67c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:00.679770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8c14a834-798b-4513-9225-75271242c67c None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:00.681507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8c14a834-798b-4513-9225-75271242c67c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:00.695793 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1213/2428] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:00.701572 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131c81d0-282d-4ca9-9aa0-bc294f2801fe None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:00.716906 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131c81d0-282d-4ca9-9aa0-bc294f2801fe None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:00.727194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c036dd56-f6eb-409d-ab42-7bc040941126 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:00.727194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c036dd56-f6eb-409d-ab42-7bc040941126 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:00.727194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c036dd56-f6eb-409d-ab42-7bc040941126 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:00.727194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c036dd56-f6eb-409d-ab42-7bc040941126 admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:00.728028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c036dd56-f6eb-409d-ab42-7bc040941126 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:56:00.728028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c036dd56-f6eb-409d-ab42-7bc040941126 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:00.729561 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c036dd56-f6eb-409d-ab42-7bc040941126 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:00.780663 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c14a834-798b-4513-9225-75271242c67c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:00.785455 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1216/2429] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:56:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:56:00.794710 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c14a834-798b-4513-9225-75271242c67c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:00.803270 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152f4353-82f7-46c4-bf4b-6fc3897c7012 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:00.804141 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152f4353-82f7-46c4-bf4b-6fc3897c7012 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:00.805020 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152f4353-82f7-46c4-bf4b-6fc3897c7012 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:00.805336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152f4353-82f7-46c4-bf4b-6fc3897c7012 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:00.806359 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152f4353-82f7-46c4-bf4b-6fc3897c7012 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:00.808617 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152f4353-82f7-46c4-bf4b-6fc3897c7012 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:00.921223 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1214/2430] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:00 2022] POST /identity/v3/projects => generated 385 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:00.922729 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c036dd56-f6eb-409d-ab42-7bc040941126 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:00.935292 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c036dd56-f6eb-409d-ab42-7bc040941126 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:01.001966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3230214a-1576-40f4-9217-6fd7dfd5bb87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:01.002604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3230214a-1576-40f4-9217-6fd7dfd5bb87 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:01.002895 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3230214a-1576-40f4-9217-6fd7dfd5bb87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:01.003223 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3230214a-1576-40f4-9217-6fd7dfd5bb87 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:01.003900 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3230214a-1576-40f4-9217-6fd7dfd5bb87 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:01.006373 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3230214a-1576-40f4-9217-6fd7dfd5bb87 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:01.127183 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1217/2431] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:00 2022] POST /identity/v3/users => generated 293 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:01.128497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152f4353-82f7-46c4-bf4b-6fc3897c7012 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:01.139593 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152f4353-82f7-46c4-bf4b-6fc3897c7012 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:01.144535 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3376bc0-a78b-41e2-b6c3-57757bd0766a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:01.155337 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3376bc0-a78b-41e2-b6c3-57757bd0766a admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:01.155337 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3376bc0-a78b-41e2-b6c3-57757bd0766a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:01.155337 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3376bc0-a78b-41e2-b6c3-57757bd0766a admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:01.155337 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3376bc0-a78b-41e2-b6c3-57757bd0766a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:01.155337 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3376bc0-a78b-41e2-b6c3-57757bd0766a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:01.168179 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1215/2432] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:01 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:01.169881 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3230214a-1576-40f4-9217-6fd7dfd5bb87 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:01.181356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3230214a-1576-40f4-9217-6fd7dfd5bb87 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:01.186424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a79004-3b41-433c-9a70-878d332a8e9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:01.187072 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a79004-3b41-433c-9a70-878d332a8e9f admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:01.187819 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a79004-3b41-433c-9a70-878d332a8e9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:01.187819 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a79004-3b41-433c-9a70-878d332a8e9f admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:01.188253 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a79004-3b41-433c-9a70-878d332a8e9f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:01.190019 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a79004-3b41-433c-9a70-878d332a8e9f admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:01.202366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1218/2433] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:01 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:01.206473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3376bc0-a78b-41e2-b6c3-57757bd0766a admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:01.218624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3376bc0-a78b-41e2-b6c3-57757bd0766a admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:01.223541 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877a0dc9-d40e-4acb-a8bd-0cb12198f2cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:01.224273 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877a0dc9-d40e-4acb-a8bd-0cb12198f2cb admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:01.224551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877a0dc9-d40e-4acb-a8bd-0cb12198f2cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:01.224892 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877a0dc9-d40e-4acb-a8bd-0cb12198f2cb admin admin] PATH_INFO: `/v3/projects/10215067d64744f7a0cca9e4abf60650/users/72466462bb8842e1bbb3aa8333becb21/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:01.239437 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-877a0dc9-d40e-4acb-a8bd-0cb12198f2cb admin admin] RBAC: Authorizing `identity:create_grant(project_id=10215067d64744f7a0cca9e4abf60650, user_id=72466462bb8842e1bbb3aa8333becb21, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:01.242656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-877a0dc9-d40e-4acb-a8bd-0cb12198f2cb admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:01.353028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a79004-3b41-433c-9a70-878d332a8e9f admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:01.356371 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d3a79004-3b41-433c-9a70-878d332a8e9f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:01.370389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a79004-3b41-433c-9a70-878d332a8e9f admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:01.375625 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1216/2434] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:01 2022] PUT /identity/v3/projects/10215067d64744f7a0cca9e4abf60650/users/72466462bb8842e1bbb3aa8333becb21/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:01.386156 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8ab644-c411-434a-8314-a121f2bbad03 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:01.390871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8ab644-c411-434a-8314-a121f2bbad03 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:01.390871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8ab644-c411-434a-8314-a121f2bbad03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:01.415707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8c8ab644-c411-434a-8314-a121f2bbad03 None None] MFA Rules not processed for user `72466462bb8842e1bbb3aa8333becb21`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:01.416928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8c8ab644-c411-434a-8314-a121f2bbad03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:01.428780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165170e9-7c73-409c-9208-456e2e2f4b72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:01.430966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165170e9-7c73-409c-9208-456e2e2f4b72 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:01.430966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165170e9-7c73-409c-9208-456e2e2f4b72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:01.430966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165170e9-7c73-409c-9208-456e2e2f4b72 admin admin] PATH_INFO: `/v3/users/ff7df9be8475405eba924b59a7bd6bb0` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:01.431989 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165170e9-7c73-409c-9208-456e2e2f4b72 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ff7df9be8475405eba924b59a7bd6bb0)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:01.434162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165170e9-7c73-409c-9208-456e2e2f4b72 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:01.457596 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=94,keystone:INSERT=3,keystone:DELETE=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:01.482220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1217/2435] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:01 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:01.493849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8ab644-c411-434a-8314-a121f2bbad03 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:01.495203 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8c8ab644-c411-434a-8314-a121f2bbad03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:01.551600 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8ab644-c411-434a-8314-a121f2bbad03 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:01.556866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0034b6aa-6e34-4d58-b0d5-3f0286767199 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:01.557472 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0034b6aa-6e34-4d58-b0d5-3f0286767199 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:01.557754 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0034b6aa-6e34-4d58-b0d5-3f0286767199 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:01.558029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0034b6aa-6e34-4d58-b0d5-3f0286767199 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:01.564547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:01.566780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0034b6aa-6e34-4d58-b0d5-3f0286767199 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:01.568240 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0034b6aa-6e34-4d58-b0d5-3f0286767199 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:01.579360 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1218/2436] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:01 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:56:01.654131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-165170e9-7c73-409c-9208-456e2e2f4b72 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ff7df9be8475405eba924b59a7bd6bb0'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:01.733661 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-165170e9-7c73-409c-9208-456e2e2f4b72 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ff7df9be8475405eba924b59a7bd6bb0'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:01.746955 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-165170e9-7c73-409c-9208-456e2e2f4b72 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ff7df9be8475405eba924b59a7bd6bb0'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:01.747616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-165170e9-7c73-409c-9208-456e2e2f4b72 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ff7df9be8475405eba924b59a7bd6bb0'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:01.754451 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-165170e9-7c73-409c-9208-456e2e2f4b72 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ff7df9be8475405eba924b59a7bd6bb0'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:01.760950 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1219/2437] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:01 2022] DELETE /identity/v3/users/ff7df9be8475405eba924b59a7bd6bb0 => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:02.020876 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0034b6aa-6e34-4d58-b0d5-3f0286767199 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:02.033420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0034b6aa-6e34-4d58-b0d5-3f0286767199 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:02.103146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f0fee0-e5c3-458d-af81-50921a1d7cd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:02.103799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f0fee0-e5c3-458d-af81-50921a1d7cd2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:02.104102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f0fee0-e5c3-458d-af81-50921a1d7cd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:02.104374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f0fee0-e5c3-458d-af81-50921a1d7cd2 admin admin] PATH_INFO: `/v3/projects/8ee4a40c6c264553a14e0eff367da4a0` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:02.105512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f0fee0-e5c3-458d-af81-50921a1d7cd2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ee4a40c6c264553a14e0eff367da4a0)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:02.107458 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f0fee0-e5c3-458d-af81-50921a1d7cd2 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:02.220195 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-99f0fee0-e5c3-458d-af81-50921a1d7cd2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ee4a40c6c264553a14e0eff367da4a0', 'request_id': 'req-99f0fee0-e5c3-458d-af81-50921a1d7cd2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:02.226493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-99f0fee0-e5c3-458d-af81-50921a1d7cd2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ee4a40c6c264553a14e0eff367da4a0', 'request_id': 'req-99f0fee0-e5c3-458d-af81-50921a1d7cd2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:02.282723 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-99f0fee0-e5c3-458d-af81-50921a1d7cd2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ee4a40c6c264553a14e0eff367da4a0', 'request_id': 'req-99f0fee0-e5c3-458d-af81-50921a1d7cd2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:02.295124 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-99f0fee0-e5c3-458d-af81-50921a1d7cd2 admin admin] The token cache is being invalidate because project 8ee4a40c6c264553a14e0eff367da4a0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:02.295416 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-99f0fee0-e5c3-458d-af81-50921a1d7cd2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:02.306221 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-99f0fee0-e5c3-458d-af81-50921a1d7cd2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:02.306221 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1219/2438] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:02 2022] DELETE /identity/v3/projects/8ee4a40c6c264553a14e0eff367da4a0 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:02.306554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9df68ac-f54e-409c-8486-00cc6a952808 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:02.306554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9df68ac-f54e-409c-8486-00cc6a952808 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:02.306554 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9df68ac-f54e-409c-8486-00cc6a952808 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:02.330008 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-f9df68ac-f54e-409c-8486-00cc6a952808 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:02.331524 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-f9df68ac-f54e-409c-8486-00cc6a952808 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:02.393225 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1220/2439] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:02.401847 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fc8e05-cdb9-4952-ac0f-ea945ad99b4c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:02.402131 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fc8e05-cdb9-4952-ac0f-ea945ad99b4c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:02.402427 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fc8e05-cdb9-4952-ac0f-ea945ad99b4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:02.423336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-b8fc8e05-cdb9-4952-ac0f-ea945ad99b4c None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:02.424829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b8fc8e05-cdb9-4952-ac0f-ea945ad99b4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:02.434390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1220/2440] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:02.441501 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9df68ac-f54e-409c-8486-00cc6a952808 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:02.451803 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9df68ac-f54e-409c-8486-00cc6a952808 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:02.456509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09502dd2-fc55-420b-b95f-399d387ebca9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:02.457108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09502dd2-fc55-420b-b95f-399d387ebca9 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:02.457471 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09502dd2-fc55-420b-b95f-399d387ebca9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:02.457753 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09502dd2-fc55-420b-b95f-399d387ebca9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:02.458382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09502dd2-fc55-420b-b95f-399d387ebca9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:56:02.458689 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09502dd2-fc55-420b-b95f-399d387ebca9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:02.460456 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09502dd2-fc55-420b-b95f-399d387ebca9 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:02.473065 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1221/2441] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:56:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:56:02.477467 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fc8e05-cdb9-4952-ac0f-ea945ad99b4c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:02.487109 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fc8e05-cdb9-4952-ac0f-ea945ad99b4c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:02.492696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ad7735-be64-45ca-a9e4-e8b109719b53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:02.493315 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ad7735-be64-45ca-a9e4-e8b109719b53 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:02.493592 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ad7735-be64-45ca-a9e4-e8b109719b53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:02.493864 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ad7735-be64-45ca-a9e4-e8b109719b53 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:02.494514 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ad7735-be64-45ca-a9e4-e8b109719b53 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:02.496452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ad7735-be64-45ca-a9e4-e8b109719b53 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:02.581242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1221/2442] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:02 2022] POST /identity/v3/projects => generated 371 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:02.585960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09502dd2-fc55-420b-b95f-399d387ebca9 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:02.596441 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09502dd2-fc55-420b-b95f-399d387ebca9 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:02.657122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed066b1-87da-498f-981b-890cfdd65649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:02.657871 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed066b1-87da-498f-981b-890cfdd65649 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:02.657992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed066b1-87da-498f-981b-890cfdd65649 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:02.658248 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed066b1-87da-498f-981b-890cfdd65649 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:02.658895 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed066b1-87da-498f-981b-890cfdd65649 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:02.660823 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed066b1-87da-498f-981b-890cfdd65649 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:02.780135 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1222/2443] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:02 2022] POST /identity/v3/users => generated 286 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:02.784154 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ad7735-be64-45ca-a9e4-e8b109719b53 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:02.794549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ad7735-be64-45ca-a9e4-e8b109719b53 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:02.799199 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e057e6-9615-4ebc-adff-ca348e9bb4ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:02.799798 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e057e6-9615-4ebc-adff-ca348e9bb4ea admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:02.800081 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e057e6-9615-4ebc-adff-ca348e9bb4ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:02.800386 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e057e6-9615-4ebc-adff-ca348e9bb4ea admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:02.800997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e057e6-9615-4ebc-adff-ca348e9bb4ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:02.802716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e057e6-9615-4ebc-adff-ca348e9bb4ea admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:02.813197 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1222/2444] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:02 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:02.817035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed066b1-87da-498f-981b-890cfdd65649 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:02.827477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed066b1-87da-498f-981b-890cfdd65649 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:02.832086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744d3896-97db-450b-8107-d6a6cf91b51d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:02.832671 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744d3896-97db-450b-8107-d6a6cf91b51d admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:02.832944 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744d3896-97db-450b-8107-d6a6cf91b51d admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:02.833213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744d3896-97db-450b-8107-d6a6cf91b51d admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:02.833843 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744d3896-97db-450b-8107-d6a6cf91b51d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:02.836480 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744d3896-97db-450b-8107-d6a6cf91b51d admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:02.846293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1223/2445] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:02 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:02.855992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e057e6-9615-4ebc-adff-ca348e9bb4ea admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:02.866118 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e057e6-9615-4ebc-adff-ca348e9bb4ea admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:02.871047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7526cbd-7ff3-4e55-bc12-83f79a2c8a2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:02.871689 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7526cbd-7ff3-4e55-bc12-83f79a2c8a2e admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:02.872766 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7526cbd-7ff3-4e55-bc12-83f79a2c8a2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:02.872766 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7526cbd-7ff3-4e55-bc12-83f79a2c8a2e admin admin] PATH_INFO: `/v3/projects/52df20a35a9d4106b3f0ff2997656bef/users/b9700eead4d24629a623f199dd92876a/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:02.887492 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7526cbd-7ff3-4e55-bc12-83f79a2c8a2e admin admin] RBAC: Authorizing `identity:create_grant(project_id=52df20a35a9d4106b3f0ff2997656bef, user_id=b9700eead4d24629a623f199dd92876a, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:02.890109 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7526cbd-7ff3-4e55-bc12-83f79a2c8a2e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:02.952290 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1223/2446] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:02 2022] PUT /identity/v3/projects/52df20a35a9d4106b3f0ff2997656bef/users/b9700eead4d24629a623f199dd92876a/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:02.963629 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0f8ceb-0268-49f1-b098-8f22f965c5fa None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:02.963913 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0f8ceb-0268-49f1-b098-8f22f965c5fa None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:02.964184 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0f8ceb-0268-49f1-b098-8f22f965c5fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:02.972593 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:02.994290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-be0f8ceb-0268-49f1-b098-8f22f965c5fa None None] MFA Rules not processed for user `b9700eead4d24629a623f199dd92876a`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:02.995624 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-be0f8ceb-0268-49f1-b098-8f22f965c5fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:03.063479 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1224/2447] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:02 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:03.067826 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7526cbd-7ff3-4e55-bc12-83f79a2c8a2e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:03.094797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:03.130668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7526cbd-7ff3-4e55-bc12-83f79a2c8a2e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:03.135998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3045fe51-8781-40c8-8ffd-8c485aeaebb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:03.136606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3045fe51-8781-40c8-8ffd-8c485aeaebb8 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:03.136888 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3045fe51-8781-40c8-8ffd-8c485aeaebb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:03.137164 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3045fe51-8781-40c8-8ffd-8c485aeaebb8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:03.137800 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3045fe51-8781-40c8-8ffd-8c485aeaebb8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:03.139745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3045fe51-8781-40c8-8ffd-8c485aeaebb8 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:03.300813 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1224/2448] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:03 2022] POST /identity/v3/projects => generated 371 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:03.305573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0f8ceb-0268-49f1-b098-8f22f965c5fa None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:03.316868 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0f8ceb-0268-49f1-b098-8f22f965c5fa None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:03.374095 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514cc86a-2e92-4f4e-ac2a-7d363c37c265 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:03.374716 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514cc86a-2e92-4f4e-ac2a-7d363c37c265 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:03.375039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514cc86a-2e92-4f4e-ac2a-7d363c37c265 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:03.375308 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514cc86a-2e92-4f4e-ac2a-7d363c37c265 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:03.376193 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514cc86a-2e92-4f4e-ac2a-7d363c37c265 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:03.378257 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514cc86a-2e92-4f4e-ac2a-7d363c37c265 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:03.483752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1225/2449] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:03 2022] POST /identity/v3/users => generated 292 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:03.488289 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3045fe51-8781-40c8-8ffd-8c485aeaebb8 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:03.501777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3045fe51-8781-40c8-8ffd-8c485aeaebb8 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:03.508926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acba428c-15aa-45c6-9b4d-66d1e6b04135 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:03.510247 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acba428c-15aa-45c6-9b4d-66d1e6b04135 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:03.510576 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acba428c-15aa-45c6-9b4d-66d1e6b04135 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:03.511114 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acba428c-15aa-45c6-9b4d-66d1e6b04135 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:03.512286 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acba428c-15aa-45c6-9b4d-66d1e6b04135 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:03.515341 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acba428c-15aa-45c6-9b4d-66d1e6b04135 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:03.533521 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1225/2450] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:03 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:03.537883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514cc86a-2e92-4f4e-ac2a-7d363c37c265 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:03.548556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514cc86a-2e92-4f4e-ac2a-7d363c37c265 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:03.553164 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6469311f-8b46-409a-b3fa-a231b4797ee6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:03.553836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6469311f-8b46-409a-b3fa-a231b4797ee6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:03.554118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6469311f-8b46-409a-b3fa-a231b4797ee6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:03.554859 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6469311f-8b46-409a-b3fa-a231b4797ee6 admin admin] PATH_INFO: `/v3/projects/42811001111a4eab8e9b12e66206075d/users/92cd8de3e606466e972033a3bae29b66/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:03.570996 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6469311f-8b46-409a-b3fa-a231b4797ee6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=42811001111a4eab8e9b12e66206075d, user_id=92cd8de3e606466e972033a3bae29b66, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:03.573661 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6469311f-8b46-409a-b3fa-a231b4797ee6 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:03.642642 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1226/2451] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:03 2022] PUT /identity/v3/projects/42811001111a4eab8e9b12e66206075d/users/92cd8de3e606466e972033a3bae29b66/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:03.646754 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acba428c-15aa-45c6-9b4d-66d1e6b04135 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:03.657788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acba428c-15aa-45c6-9b4d-66d1e6b04135 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:03.715065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc2b7d8-3fb0-4567-b8cf-ecc1960e7b03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:03.715669 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc2b7d8-3fb0-4567-b8cf-ecc1960e7b03 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:03.715976 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc2b7d8-3fb0-4567-b8cf-ecc1960e7b03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:03.716247 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc2b7d8-3fb0-4567-b8cf-ecc1960e7b03 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:03.716882 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc2b7d8-3fb0-4567-b8cf-ecc1960e7b03 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:03.718443 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc2b7d8-3fb0-4567-b8cf-ecc1960e7b03 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:03.728015 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1226/2452] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:03 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:03.732372 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6469311f-8b46-409a-b3fa-a231b4797ee6 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:03.746172 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6469311f-8b46-409a-b3fa-a231b4797ee6 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:03.749324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb597fcf-4ec3-4d19-ad4d-6c83579b9fde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:03.750152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb597fcf-4ec3-4d19-ad4d-6c83579b9fde admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:03.750297 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb597fcf-4ec3-4d19-ad4d-6c83579b9fde admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:03.750518 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb597fcf-4ec3-4d19-ad4d-6c83579b9fde admin admin] PATH_INFO: `/v3/projects/42811001111a4eab8e9b12e66206075d/users/92cd8de3e606466e972033a3bae29b66/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:03.752817 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb597fcf-4ec3-4d19-ad4d-6c83579b9fde admin admin] RBAC: Authorizing `identity:create_grant(project_id=42811001111a4eab8e9b12e66206075d, user_id=92cd8de3e606466e972033a3bae29b66, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:03.755575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb597fcf-4ec3-4d19-ad4d-6c83579b9fde admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:03.768176 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1227/2453] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:03 2022] PUT /identity/v3/projects/42811001111a4eab8e9b12e66206075d/users/92cd8de3e606466e972033a3bae29b66/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:03.782219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c97e53-616f-4450-989c-91b706de39d9 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:03.782509 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c97e53-616f-4450-989c-91b706de39d9 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:03.782796 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c97e53-616f-4450-989c-91b706de39d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:03.812684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-51c97e53-616f-4450-989c-91b706de39d9 None None] MFA Rules not processed for user `92cd8de3e606466e972033a3bae29b66`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:03.813955 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-51c97e53-616f-4450-989c-91b706de39d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:03.886436 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1227/2454] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:03 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:03.897532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb597fcf-4ec3-4d19-ad4d-6c83579b9fde admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:03.900256 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-cb597fcf-4ec3-4d19-ad4d-6c83579b9fde admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:03.953083 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:03.958716 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb597fcf-4ec3-4d19-ad4d-6c83579b9fde admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:03.962656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69020537-f891-46aa-9863-04a92cc01570 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:03.963297 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69020537-f891-46aa-9863-04a92cc01570 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:03.963583 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69020537-f891-46aa-9863-04a92cc01570 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:03.963859 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69020537-f891-46aa-9863-04a92cc01570 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:03.972771 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69020537-f891-46aa-9863-04a92cc01570 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:03.974200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69020537-f891-46aa-9863-04a92cc01570 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:03.983693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1228/2455] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:03 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:04.268501 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c97e53-616f-4450-989c-91b706de39d9 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:04.269980 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-51c97e53-616f-4450-989c-91b706de39d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:04.291204 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:04.342670 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c97e53-616f-4450-989c-91b706de39d9 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:04.347540 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128b5412-8021-42c0-a4db-dffe6ae7de35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:04.352054 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128b5412-8021-42c0-a4db-dffe6ae7de35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:04.352054 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128b5412-8021-42c0-a4db-dffe6ae7de35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:04.352054 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128b5412-8021-42c0-a4db-dffe6ae7de35 admin admin] PATH_INFO: `/v3/users/e7467dbdeede4c2ebafe9a255d48939d` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:04.352054 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128b5412-8021-42c0-a4db-dffe6ae7de35 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e7467dbdeede4c2ebafe9a255d48939d)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:04.352812 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128b5412-8021-42c0-a4db-dffe6ae7de35 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:04.490743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-128b5412-8021-42c0-a4db-dffe6ae7de35 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e7467dbdeede4c2ebafe9a255d48939d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:04.503311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-128b5412-8021-42c0-a4db-dffe6ae7de35 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e7467dbdeede4c2ebafe9a255d48939d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:04.578936 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-128b5412-8021-42c0-a4db-dffe6ae7de35 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e7467dbdeede4c2ebafe9a255d48939d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:04.579638 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-128b5412-8021-42c0-a4db-dffe6ae7de35 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e7467dbdeede4c2ebafe9a255d48939d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:04.587789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-128b5412-8021-42c0-a4db-dffe6ae7de35 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e7467dbdeede4c2ebafe9a255d48939d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:04.593703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1228/2456] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:04 2022] DELETE /identity/v3/users/e7467dbdeede4c2ebafe9a255d48939d => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:04.598351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15109e83-ffdf-4ca7-b48b-42a12250fb0a None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:04.598638 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15109e83-ffdf-4ca7-b48b-42a12250fb0a None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:04.598969 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15109e83-ffdf-4ca7-b48b-42a12250fb0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:04.617734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-15109e83-ffdf-4ca7-b48b-42a12250fb0a None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:04.620937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-15109e83-ffdf-4ca7-b48b-42a12250fb0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:04.678627 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1229/2457] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:04.689283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128b5412-8021-42c0-a4db-dffe6ae7de35 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:04.690702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-128b5412-8021-42c0-a4db-dffe6ae7de35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:04.750640 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128b5412-8021-42c0-a4db-dffe6ae7de35 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:04.754308 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8f5683-f740-420e-9812-2f6f0864b4fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:04.754932 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8f5683-f740-420e-9812-2f6f0864b4fb service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:04.755342 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8f5683-f740-420e-9812-2f6f0864b4fb service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:04.755625 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8f5683-f740-420e-9812-2f6f0864b4fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:04.765490 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8f5683-f740-420e-9812-2f6f0864b4fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:04.770597 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8f5683-f740-420e-9812-2f6f0864b4fb service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:04.777523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1229/2458] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:04.835131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15109e83-ffdf-4ca7-b48b-42a12250fb0a None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:04.837096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-15109e83-ffdf-4ca7-b48b-42a12250fb0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:04.849072 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15109e83-ffdf-4ca7-b48b-42a12250fb0a None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:04.855458 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3560051-47d6-49a6-bc9f-9b9caa9f3244 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:04.856244 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3560051-47d6-49a6-bc9f-9b9caa9f3244 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:04.856534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3560051-47d6-49a6-bc9f-9b9caa9f3244 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:04.858818 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3560051-47d6-49a6-bc9f-9b9caa9f3244 admin admin] PATH_INFO: `/v3/users/72466462bb8842e1bbb3aa8333becb21` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:04.859501 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3560051-47d6-49a6-bc9f-9b9caa9f3244 admin admin] RBAC: Authorizing `identity:delete_user(user_id=72466462bb8842e1bbb3aa8333becb21)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:04.862105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3560051-47d6-49a6-bc9f-9b9caa9f3244 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:04.951233 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8f5683-f740-420e-9812-2f6f0864b4fb service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:04.952703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8a8f5683-f740-420e-9812-2f6f0864b4fb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:04.966172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8f5683-f740-420e-9812-2f6f0864b4fb service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:04.970825 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9db16cb-d834-4f6d-a33e-92ccb76f67d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:04.971473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9db16cb-d834-4f6d-a33e-92ccb76f67d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:04.971751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9db16cb-d834-4f6d-a33e-92ccb76f67d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:04.972106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9db16cb-d834-4f6d-a33e-92ccb76f67d5 admin admin] PATH_INFO: `/v3/users/b9700eead4d24629a623f199dd92876a` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:04.973686 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9db16cb-d834-4f6d-a33e-92ccb76f67d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b9700eead4d24629a623f199dd92876a)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:04.975711 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9db16cb-d834-4f6d-a33e-92ccb76f67d5 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:05.092599 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-b3560051-47d6-49a6-bc9f-9b9caa9f3244 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '72466462bb8842e1bbb3aa8333becb21'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:05.156431 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:05.167532 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b9db16cb-d834-4f6d-a33e-92ccb76f67d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b9700eead4d24629a623f199dd92876a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:05.180032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b9db16cb-d834-4f6d-a33e-92ccb76f67d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b9700eead4d24629a623f199dd92876a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:05.207933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-b3560051-47d6-49a6-bc9f-9b9caa9f3244 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '72466462bb8842e1bbb3aa8333becb21'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:05.220559 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-b3560051-47d6-49a6-bc9f-9b9caa9f3244 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '72466462bb8842e1bbb3aa8333becb21'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:05.221206 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-b3560051-47d6-49a6-bc9f-9b9caa9f3244 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '72466462bb8842e1bbb3aa8333becb21'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:05.228669 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-b3560051-47d6-49a6-bc9f-9b9caa9f3244 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '72466462bb8842e1bbb3aa8333becb21'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:05.234543 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1230/2459] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:04 2022] DELETE /identity/v3/users/72466462bb8842e1bbb3aa8333becb21 => generated 0 bytes in 400 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:05.238978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91985d0d-812a-482b-b737-a48cefac0c9f None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:05.239271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91985d0d-812a-482b-b737-a48cefac0c9f None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:05.239553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91985d0d-812a-482b-b737-a48cefac0c9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:05.259184 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-91985d0d-812a-482b-b737-a48cefac0c9f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:05.260364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-91985d0d-812a-482b-b737-a48cefac0c9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:05.306294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b9db16cb-d834-4f6d-a33e-92ccb76f67d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b9700eead4d24629a623f199dd92876a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:05.308798 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b9db16cb-d834-4f6d-a33e-92ccb76f67d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b9700eead4d24629a623f199dd92876a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:05.317089 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b9db16cb-d834-4f6d-a33e-92ccb76f67d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b9700eead4d24629a623f199dd92876a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:05.331983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1230/2460] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:04 2022] DELETE /identity/v3/users/b9700eead4d24629a623f199dd92876a => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:56:05.331983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9db16cb-d834-4f6d-a33e-92ccb76f67d5 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:05.331983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b9db16cb-d834-4f6d-a33e-92ccb76f67d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:05.334500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1231/2461] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:56:05.335966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6619aab-c7ce-49bb-a42e-0b2472cc8951 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:05.336253 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6619aab-c7ce-49bb-a42e-0b2472cc8951 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:05.336531 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6619aab-c7ce-49bb-a42e-0b2472cc8951 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:05.343692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:05.358000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-e6619aab-c7ce-49bb-a42e-0b2472cc8951 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:05.359262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e6619aab-c7ce-49bb-a42e-0b2472cc8951 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:05.402792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9db16cb-d834-4f6d-a33e-92ccb76f67d5 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:05.407797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e644ea-7bf5-4777-bb7c-b027a3b49252 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:05.408429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e644ea-7bf5-4777-bb7c-b027a3b49252 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:05.408712 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e644ea-7bf5-4777-bb7c-b027a3b49252 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:05.408982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e644ea-7bf5-4777-bb7c-b027a3b49252 admin admin] PATH_INFO: `/v3/projects/a698a207b126483ab7659f5dc9e280e8` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:05.410096 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e644ea-7bf5-4777-bb7c-b027a3b49252 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a698a207b126483ab7659f5dc9e280e8)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:05.412035 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e644ea-7bf5-4777-bb7c-b027a3b49252 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:05.421361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1232/2462] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:05.422389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6619aab-c7ce-49bb-a42e-0b2472cc8951 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:05.424031 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e6619aab-c7ce-49bb-a42e-0b2472cc8951 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:05.481284 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6619aab-c7ce-49bb-a42e-0b2472cc8951 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:05.493481 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-70e644ea-7bf5-4777-bb7c-b027a3b49252 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a698a207b126483ab7659f5dc9e280e8', 'request_id': 'req-70e644ea-7bf5-4777-bb7c-b027a3b49252'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:05.499738 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-70e644ea-7bf5-4777-bb7c-b027a3b49252 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a698a207b126483ab7659f5dc9e280e8', 'request_id': 'req-70e644ea-7bf5-4777-bb7c-b027a3b49252'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:05.529928 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb42833-2d02-4bcd-b17a-32985e618b3f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:05.530534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb42833-2d02-4bcd-b17a-32985e618b3f service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:05.530812 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb42833-2d02-4bcd-b17a-32985e618b3f service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:05.531110 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb42833-2d02-4bcd-b17a-32985e618b3f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:05.573718 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-70e644ea-7bf5-4777-bb7c-b027a3b49252 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a698a207b126483ab7659f5dc9e280e8', 'request_id': 'req-70e644ea-7bf5-4777-bb7c-b027a3b49252'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:05.584744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-70e644ea-7bf5-4777-bb7c-b027a3b49252 admin admin] The token cache is being invalidate because project a698a207b126483ab7659f5dc9e280e8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:05.585042 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-70e644ea-7bf5-4777-bb7c-b027a3b49252 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:05.586419 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-70e644ea-7bf5-4777-bb7c-b027a3b49252 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:05.587967 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1231/2463] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:05 2022] DELETE /identity/v3/projects/a698a207b126483ab7659f5dc9e280e8 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:05.588980 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e644ea-7bf5-4777-bb7c-b027a3b49252 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:05.590453 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb42833-2d02-4bcd-b17a-32985e618b3f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:05.591944 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb42833-2d02-4bcd-b17a-32985e618b3f service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:05.595369 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-70e644ea-7bf5-4777-bb7c-b027a3b49252 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:05.602805 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1233/2464] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:05.604285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987577a2-f81f-4b0e-82dc-42277ff10f51 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:05.604578 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987577a2-f81f-4b0e-82dc-42277ff10f51 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:05.605648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987577a2-f81f-4b0e-82dc-42277ff10f51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:05.608994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e644ea-7bf5-4777-bb7c-b027a3b49252 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:05.612806 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ce6cce-2289-4183-bf86-5f0a4f8a3100 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:05.613494 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ce6cce-2289-4183-bf86-5f0a4f8a3100 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:05.613781 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ce6cce-2289-4183-bf86-5f0a4f8a3100 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:05.614054 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ce6cce-2289-4183-bf86-5f0a4f8a3100 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:05.615849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-25ce6cce-2289-4183-bf86-5f0a4f8a3100 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:05.628080 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-987577a2-f81f-4b0e-82dc-42277ff10f51 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:05.629395 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-987577a2-f81f-4b0e-82dc-42277ff10f51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:05.629879 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ce6cce-2289-4183-bf86-5f0a4f8a3100 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:05.631462 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ce6cce-2289-4183-bf86-5f0a4f8a3100 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:05.638734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1234/2465] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:05.642312 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1232/2466] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:05.646636 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157c828a-fc61-446b-8fc3-f25689d27d55 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:05.647007 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157c828a-fc61-446b-8fc3-f25689d27d55 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:05.647375 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157c828a-fc61-446b-8fc3-f25689d27d55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:05.671745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-157c828a-fc61-446b-8fc3-f25689d27d55 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:05.673013 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-157c828a-fc61-446b-8fc3-f25689d27d55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:05.685807 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1235/2467] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:05.686525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ce6cce-2289-4183-bf86-5f0a4f8a3100 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:05.696904 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ce6cce-2289-4183-bf86-5f0a4f8a3100 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:05.701783 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8390e57-8d9a-4dea-968f-f173204f0808 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:05.705292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8390e57-8d9a-4dea-968f-f173204f0808 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:05.705292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8390e57-8d9a-4dea-968f-f173204f0808 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:05.705292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8390e57-8d9a-4dea-968f-f173204f0808 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:05.705292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8390e57-8d9a-4dea-968f-f173204f0808 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:56:05.705292 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8390e57-8d9a-4dea-968f-f173204f0808 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:05.705725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8390e57-8d9a-4dea-968f-f173204f0808 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:05.721790 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1233/2468] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:56:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:56:05.723079 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157c828a-fc61-446b-8fc3-f25689d27d55 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:05.733451 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157c828a-fc61-446b-8fc3-f25689d27d55 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:05.744420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24980f8c-9028-4749-b6fc-63312c084cdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:05.744420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24980f8c-9028-4749-b6fc-63312c084cdc admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:05.744420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24980f8c-9028-4749-b6fc-63312c084cdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:05.744420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24980f8c-9028-4749-b6fc-63312c084cdc admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:05.745280 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24980f8c-9028-4749-b6fc-63312c084cdc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:05.745280 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24980f8c-9028-4749-b6fc-63312c084cdc admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:05.763942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:05.868888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1236/2469] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:05 2022] POST /identity/v3/projects => generated 365 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:05.873856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8390e57-8d9a-4dea-968f-f173204f0808 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:05.884550 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8390e57-8d9a-4dea-968f-f173204f0808 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:05.896982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:05.947045 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1f87e2-4551-4af1-89a9-2aacbc099143 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:05.947670 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1f87e2-4551-4af1-89a9-2aacbc099143 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:05.947954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1f87e2-4551-4af1-89a9-2aacbc099143 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:05.948235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1f87e2-4551-4af1-89a9-2aacbc099143 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:05.948878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1f87e2-4551-4af1-89a9-2aacbc099143 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:05.950995 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1f87e2-4551-4af1-89a9-2aacbc099143 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:06.034884 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1234/2470] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:05 2022] POST /identity/v3/users => generated 283 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:06.036278 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24980f8c-9028-4749-b6fc-63312c084cdc admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:06.046751 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24980f8c-9028-4749-b6fc-63312c084cdc admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:06.051544 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db75d74-bb87-474f-8271-df4f80c976bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:06.056217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db75d74-bb87-474f-8271-df4f80c976bc admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:06.056217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db75d74-bb87-474f-8271-df4f80c976bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:06.056217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db75d74-bb87-474f-8271-df4f80c976bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:06.056217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db75d74-bb87-474f-8271-df4f80c976bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:06.056217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db75d74-bb87-474f-8271-df4f80c976bc admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:06.065007 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1237/2471] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:06 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:06.069092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1f87e2-4551-4af1-89a9-2aacbc099143 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:06.079443 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1f87e2-4551-4af1-89a9-2aacbc099143 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:06.084087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d6fdf2-ec16-449a-8548-fb3edafe8c32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:06.084615 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d6fdf2-ec16-449a-8548-fb3edafe8c32 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:06.084908 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d6fdf2-ec16-449a-8548-fb3edafe8c32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:06.085165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d6fdf2-ec16-449a-8548-fb3edafe8c32 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:06.086568 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d6fdf2-ec16-449a-8548-fb3edafe8c32 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:06.088178 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d6fdf2-ec16-449a-8548-fb3edafe8c32 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:06.095294 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db75d74-bb87-474f-8271-df4f80c976bc admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:06.097562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3db75d74-bb87-474f-8271-df4f80c976bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:06.099592 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1235/2472] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:06 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:06.103903 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d6fdf2-ec16-449a-8548-fb3edafe8c32 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:06.114330 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d6fdf2-ec16-449a-8548-fb3edafe8c32 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:06.117914 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db75d74-bb87-474f-8271-df4f80c976bc admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:06.122724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c62e57a-1ebf-4edf-80b5-c7eb22aa3687 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:06.126008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c62e57a-1ebf-4edf-80b5-c7eb22aa3687 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:06.126008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c62e57a-1ebf-4edf-80b5-c7eb22aa3687 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:06.126008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c62e57a-1ebf-4edf-80b5-c7eb22aa3687 admin admin] PATH_INFO: `/v3/projects/db87e234bcbb4da7923ff645da7ceed5/users/814784d411e9463abeb972279b92e4f8/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:06.130165 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714cc5b1-8271-46ee-9308-e9eccd2d0a1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:06.130815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714cc5b1-8271-46ee-9308-e9eccd2d0a1d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:06.131126 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714cc5b1-8271-46ee-9308-e9eccd2d0a1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:06.131453 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714cc5b1-8271-46ee-9308-e9eccd2d0a1d admin admin] PATH_INFO: `/v3/projects/10215067d64744f7a0cca9e4abf60650` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:06.133296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714cc5b1-8271-46ee-9308-e9eccd2d0a1d admin admin] RBAC: Authorizing `identity:delete_project(project_id=10215067d64744f7a0cca9e4abf60650)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:06.136345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714cc5b1-8271-46ee-9308-e9eccd2d0a1d admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:06.151190 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c62e57a-1ebf-4edf-80b5-c7eb22aa3687 admin admin] RBAC: Authorizing `identity:create_grant(project_id=db87e234bcbb4da7923ff645da7ceed5, user_id=814784d411e9463abeb972279b92e4f8, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:06.154767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c62e57a-1ebf-4edf-80b5-c7eb22aa3687 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:06.254403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1236/2473] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:06 2022] PUT /identity/v3/projects/db87e234bcbb4da7923ff645da7ceed5/users/814784d411e9463abeb972279b92e4f8/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:56:06.255397 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c62e57a-1ebf-4edf-80b5-c7eb22aa3687 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:06.256983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7c62e57a-1ebf-4edf-80b5-c7eb22aa3687 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:06.324441 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c62e57a-1ebf-4edf-80b5-c7eb22aa3687 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:06.329741 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8c87e2-e249-4db8-b7e9-c593eaea0d01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:06.330374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8c87e2-e249-4db8-b7e9-c593eaea0d01 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:06.330655 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8c87e2-e249-4db8-b7e9-c593eaea0d01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:06.330943 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8c87e2-e249-4db8-b7e9-c593eaea0d01 admin admin] PATH_INFO: `/v3/projects/52df20a35a9d4106b3f0ff2997656bef` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:06.332103 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8c87e2-e249-4db8-b7e9-c593eaea0d01 admin admin] RBAC: Authorizing `identity:delete_project(project_id=52df20a35a9d4106b3f0ff2997656bef)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:06.333986 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8c87e2-e249-4db8-b7e9-c593eaea0d01 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:06.341650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-714cc5b1-8271-46ee-9308-e9eccd2d0a1d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '10215067d64744f7a0cca9e4abf60650', 'request_id': 'req-714cc5b1-8271-46ee-9308-e9eccd2d0a1d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:06.470932 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-714cc5b1-8271-46ee-9308-e9eccd2d0a1d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '10215067d64744f7a0cca9e4abf60650', 'request_id': 'req-714cc5b1-8271-46ee-9308-e9eccd2d0a1d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:06.477845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-714cc5b1-8271-46ee-9308-e9eccd2d0a1d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '10215067d64744f7a0cca9e4abf60650', 'request_id': 'req-714cc5b1-8271-46ee-9308-e9eccd2d0a1d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:06.489870 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-714cc5b1-8271-46ee-9308-e9eccd2d0a1d admin admin] The token cache is being invalidate because project 10215067d64744f7a0cca9e4abf60650 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:06.490160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-714cc5b1-8271-46ee-9308-e9eccd2d0a1d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:06.490805 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-714cc5b1-8271-46ee-9308-e9eccd2d0a1d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:06.492922 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1238/2474] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:06 2022] DELETE /identity/v3/projects/10215067d64744f7a0cca9e4abf60650 => generated 0 bytes in 399 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:56:06.502348 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2f8c87e2-e249-4db8-b7e9-c593eaea0d01 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '52df20a35a9d4106b3f0ff2997656bef', 'request_id': 'req-2f8c87e2-e249-4db8-b7e9-c593eaea0d01'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:06.505776 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2de25f5-e0ac-4bdc-88cc-fccdf779f707 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:06.506080 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2de25f5-e0ac-4bdc-88cc-fccdf779f707 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:06.506556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2de25f5-e0ac-4bdc-88cc-fccdf779f707 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:06.511136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2f8c87e2-e249-4db8-b7e9-c593eaea0d01 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '52df20a35a9d4106b3f0ff2997656bef', 'request_id': 'req-2f8c87e2-e249-4db8-b7e9-c593eaea0d01'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:06.541278 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-d2de25f5-e0ac-4bdc-88cc-fccdf779f707 None None] MFA Rules not processed for user `814784d411e9463abeb972279b92e4f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:06.542524 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d2de25f5-e0ac-4bdc-88cc-fccdf779f707 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:06.596215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2f8c87e2-e249-4db8-b7e9-c593eaea0d01 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '52df20a35a9d4106b3f0ff2997656bef', 'request_id': 'req-2f8c87e2-e249-4db8-b7e9-c593eaea0d01'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:06.608451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:06.613281 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2f8c87e2-e249-4db8-b7e9-c593eaea0d01 admin admin] The token cache is being invalidate because project 52df20a35a9d4106b3f0ff2997656bef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:06.613605 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2f8c87e2-e249-4db8-b7e9-c593eaea0d01 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:06.614279 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2f8c87e2-e249-4db8-b7e9-c593eaea0d01 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:06.616265 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1237/2475] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:06 2022] DELETE /identity/v3/projects/52df20a35a9d4106b3f0ff2997656bef => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:06.619271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1239/2476] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:06 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:06.620731 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8c87e2-e249-4db8-b7e9-c593eaea0d01 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:06.622370 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2f8c87e2-e249-4db8-b7e9-c593eaea0d01 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:06.623401 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2de25f5-e0ac-4bdc-88cc-fccdf779f707 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:06.624771 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d2de25f5-e0ac-4bdc-88cc-fccdf779f707 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:06.672174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:06.695802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8c87e2-e249-4db8-b7e9-c593eaea0d01 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:06.701105 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f68aa7-19af-4649-a1f0-681684ea88f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:06.706081 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f68aa7-19af-4649-a1f0-681684ea88f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:06.706081 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f68aa7-19af-4649-a1f0-681684ea88f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:06.706081 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f68aa7-19af-4649-a1f0-681684ea88f3 admin admin] PATH_INFO: `/v3/users/92cd8de3e606466e972033a3bae29b66` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:06.706542 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2de25f5-e0ac-4bdc-88cc-fccdf779f707 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:06.706648 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f68aa7-19af-4649-a1f0-681684ea88f3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=92cd8de3e606466e972033a3bae29b66)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:06.706648 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f68aa7-19af-4649-a1f0-681684ea88f3 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:06.708632 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e7904f-f67a-46e8-9ac3-50e4e5e8af8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:06.709261 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e7904f-f67a-46e8-9ac3-50e4e5e8af8f admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:06.709561 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e7904f-f67a-46e8-9ac3-50e4e5e8af8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:06.709814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e7904f-f67a-46e8-9ac3-50e4e5e8af8f admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:06.711465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e7904f-f67a-46e8-9ac3-50e4e5e8af8f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:06.713585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e7904f-f67a-46e8-9ac3-50e4e5e8af8f admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:06.872832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1240/2477] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:06 2022] POST /identity/v3/projects => generated 363 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:56:06.877283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e7904f-f67a-46e8-9ac3-50e4e5e8af8f admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:06.878995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-f5e7904f-f67a-46e8-9ac3-50e4e5e8af8f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:06.942505 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e7904f-f67a-46e8-9ac3-50e4e5e8af8f admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:06.947971 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a3750e-3d34-48d4-9b02-56f1e55e75b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:06.948572 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a3750e-3d34-48d4-9b02-56f1e55e75b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:06.948858 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a3750e-3d34-48d4-9b02-56f1e55e75b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:06.949121 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a3750e-3d34-48d4-9b02-56f1e55e75b9 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:06.949753 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a3750e-3d34-48d4-9b02-56f1e55e75b9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:06.951699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a3750e-3d34-48d4-9b02-56f1e55e75b9 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:06.954246 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-29f68aa7-19af-4649-a1f0-681684ea88f3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '92cd8de3e606466e972033a3bae29b66'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:06.967028 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-29f68aa7-19af-4649-a1f0-681684ea88f3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '92cd8de3e606466e972033a3bae29b66'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:07.053301 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-29f68aa7-19af-4649-a1f0-681684ea88f3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '92cd8de3e606466e972033a3bae29b66'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:07.054557 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-29f68aa7-19af-4649-a1f0-681684ea88f3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '92cd8de3e606466e972033a3bae29b66'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:07.060208 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1241/2478] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:06 2022] POST /identity/v3/users => generated 288 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:56:07.064096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a3750e-3d34-48d4-9b02-56f1e55e75b9 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:07.065560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-29f68aa7-19af-4649-a1f0-681684ea88f3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '92cd8de3e606466e972033a3bae29b66'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:07.072630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1238/2479] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:06 2022] DELETE /identity/v3/users/92cd8de3e606466e972033a3bae29b66 => generated 0 bytes in 452 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:07.074460 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a3750e-3d34-48d4-9b02-56f1e55e75b9 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:07.131298 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aef2136-583f-4041-b2c5-6a9c19989502 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:07.131835 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aef2136-583f-4041-b2c5-6a9c19989502 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:07.132773 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aef2136-583f-4041-b2c5-6a9c19989502 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:07.133062 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aef2136-583f-4041-b2c5-6a9c19989502 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:07.133703 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aef2136-583f-4041-b2c5-6a9c19989502 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:07.135402 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aef2136-583f-4041-b2c5-6a9c19989502 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:07.145063 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1242/2480] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:07 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:56:07.149422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f68aa7-19af-4649-a1f0-681684ea88f3 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:07.150832 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-29f68aa7-19af-4649-a1f0-681684ea88f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:07.164029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f68aa7-19af-4649-a1f0-681684ea88f3 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:07.168651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7020bb3c-67b7-43cc-8ddc-058546842963 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:07.169351 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7020bb3c-67b7-43cc-8ddc-058546842963 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:07.169680 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7020bb3c-67b7-43cc-8ddc-058546842963 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:07.169956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7020bb3c-67b7-43cc-8ddc-058546842963 admin admin] PATH_INFO: `/v3/projects/ccc66f18155d46c1ba4dc93f4f69a28f/users/9b291a82a59945fbb5c82e1925798157/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:07.185453 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7020bb3c-67b7-43cc-8ddc-058546842963 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ccc66f18155d46c1ba4dc93f4f69a28f, user_id=9b291a82a59945fbb5c82e1925798157, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:07.188132 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7020bb3c-67b7-43cc-8ddc-058546842963 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:07.270568 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1239/2481] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:07 2022] PUT /identity/v3/projects/ccc66f18155d46c1ba4dc93f4f69a28f/users/9b291a82a59945fbb5c82e1925798157/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:07.274740 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aef2136-583f-4041-b2c5-6a9c19989502 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:07.285160 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aef2136-583f-4041-b2c5-6a9c19989502 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:07.339845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:07.353028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf520e7-52f3-4456-a327-8faeb1d6f1ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:07.353637 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf520e7-52f3-4456-a327-8faeb1d6f1ba admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:07.353922 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf520e7-52f3-4456-a327-8faeb1d6f1ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:07.354200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf520e7-52f3-4456-a327-8faeb1d6f1ba admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:07.354837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf520e7-52f3-4456-a327-8faeb1d6f1ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:07.356473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf520e7-52f3-4456-a327-8faeb1d6f1ba admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:07.366023 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1243/2482] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:07 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:07.370875 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7020bb3c-67b7-43cc-8ddc-058546842963 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:07.386725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7020bb3c-67b7-43cc-8ddc-058546842963 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:07.386725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a396f4a2-7a5c-4fb8-9c82-b78023518ab6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:07.386725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a396f4a2-7a5c-4fb8-9c82-b78023518ab6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:07.387092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a396f4a2-7a5c-4fb8-9c82-b78023518ab6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:07.387185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a396f4a2-7a5c-4fb8-9c82-b78023518ab6 admin admin] PATH_INFO: `/v3/projects/ccc66f18155d46c1ba4dc93f4f69a28f/users/9b291a82a59945fbb5c82e1925798157/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:07.389030 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a396f4a2-7a5c-4fb8-9c82-b78023518ab6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ccc66f18155d46c1ba4dc93f4f69a28f, user_id=9b291a82a59945fbb5c82e1925798157, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:07.391669 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a396f4a2-7a5c-4fb8-9c82-b78023518ab6 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:07.401679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1240/2483] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:07 2022] PUT /identity/v3/projects/ccc66f18155d46c1ba4dc93f4f69a28f/users/9b291a82a59945fbb5c82e1925798157/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:07.411763 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f28776-4b8f-40c6-8267-1b78b666967a None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:07.412055 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f28776-4b8f-40c6-8267-1b78b666967a None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:07.412334 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f28776-4b8f-40c6-8267-1b78b666967a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:07.441140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-70f28776-4b8f-40c6-8267-1b78b666967a None None] MFA Rules not processed for user `9b291a82a59945fbb5c82e1925798157`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:07.443010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-70f28776-4b8f-40c6-8267-1b78b666967a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:07.522676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1244/2484] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:07 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:07.534705 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a396f4a2-7a5c-4fb8-9c82-b78023518ab6 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:07.536463 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a396f4a2-7a5c-4fb8-9c82-b78023518ab6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:07.600992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=87,keystone:DELETE=10,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:07.614099 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a396f4a2-7a5c-4fb8-9c82-b78023518ab6 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:07.619175 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0488b0b4-b7a7-4e80-b06f-c6399248bdf0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:07.619792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0488b0b4-b7a7-4e80-b06f-c6399248bdf0 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:07.620074 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0488b0b4-b7a7-4e80-b06f-c6399248bdf0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:07.620472 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0488b0b4-b7a7-4e80-b06f-c6399248bdf0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:07.630328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0488b0b4-b7a7-4e80-b06f-c6399248bdf0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:07.631820 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0488b0b4-b7a7-4e80-b06f-c6399248bdf0 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:07.643590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1241/2485] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:07 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:07.702263 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f28776-4b8f-40c6-8267-1b78b666967a None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:07.705688 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-70f28776-4b8f-40c6-8267-1b78b666967a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:07.783325 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f28776-4b8f-40c6-8267-1b78b666967a None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:07.788688 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a994a714-8b98-4e3f-8da1-bfe20e0451ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:07.789244 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a994a714-8b98-4e3f-8da1-bfe20e0451ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:07.789586 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a994a714-8b98-4e3f-8da1-bfe20e0451ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:07.789965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a994a714-8b98-4e3f-8da1-bfe20e0451ac admin admin] PATH_INFO: `/v3/projects/42811001111a4eab8e9b12e66206075d` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:07.791133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a994a714-8b98-4e3f-8da1-bfe20e0451ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=42811001111a4eab8e9b12e66206075d)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:07.793030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a994a714-8b98-4e3f-8da1-bfe20e0451ac admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:07.930991 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0488b0b4-b7a7-4e80-b06f-c6399248bdf0 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:07.939839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0488b0b4-b7a7-4e80-b06f-c6399248bdf0 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:07.943420 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d091f179-64d1-416d-bdbb-80eb650aadf1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:07.944007 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d091f179-64d1-416d-bdbb-80eb650aadf1 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:07.944289 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d091f179-64d1-416d-bdbb-80eb650aadf1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:07.944569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d091f179-64d1-416d-bdbb-80eb650aadf1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:07.947025 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d091f179-64d1-416d-bdbb-80eb650aadf1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:07.968383 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a994a714-8b98-4e3f-8da1-bfe20e0451ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '42811001111a4eab8e9b12e66206075d', 'request_id': 'req-a994a714-8b98-4e3f-8da1-bfe20e0451ac'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:08.008390 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d091f179-64d1-416d-bdbb-80eb650aadf1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:08.009840 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d091f179-64d1-416d-bdbb-80eb650aadf1 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:08.018608 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1242/2486] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:07 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:56:08.053344 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a994a714-8b98-4e3f-8da1-bfe20e0451ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '42811001111a4eab8e9b12e66206075d', 'request_id': 'req-a994a714-8b98-4e3f-8da1-bfe20e0451ac'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:08.060740 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a994a714-8b98-4e3f-8da1-bfe20e0451ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '42811001111a4eab8e9b12e66206075d', 'request_id': 'req-a994a714-8b98-4e3f-8da1-bfe20e0451ac'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:08.068157 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:08.074104 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a994a714-8b98-4e3f-8da1-bfe20e0451ac admin admin] The token cache is being invalidate because project 42811001111a4eab8e9b12e66206075d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:08.074104 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a994a714-8b98-4e3f-8da1-bfe20e0451ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:08.074104 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a994a714-8b98-4e3f-8da1-bfe20e0451ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:08.075151 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1245/2487] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:07 2022] DELETE /identity/v3/projects/42811001111a4eab8e9b12e66206075d => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:08.080026 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa924330-6c18-47ea-bbed-805b045a06e8 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:08.080339 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa924330-6c18-47ea-bbed-805b045a06e8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:08.080614 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa924330-6c18-47ea-bbed-805b045a06e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:08.100270 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-aa924330-6c18-47ea-bbed-805b045a06e8 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:08.101542 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-aa924330-6c18-47ea-bbed-805b045a06e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:08.163842 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1243/2488] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:08.171971 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412d64cc-736a-4adf-990e-f24d6b11dab2 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:08.172174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412d64cc-736a-4adf-990e-f24d6b11dab2 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:08.172447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412d64cc-736a-4adf-990e-f24d6b11dab2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:08.192533 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-412d64cc-736a-4adf-990e-f24d6b11dab2 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:08.193713 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-412d64cc-736a-4adf-990e-f24d6b11dab2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:08.201855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1246/2489] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:08.205922 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa924330-6c18-47ea-bbed-805b045a06e8 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:08.215827 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa924330-6c18-47ea-bbed-805b045a06e8 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:08.220591 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30fc0c2-cff5-4cba-a5c8-3536a3b42d1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:08.228942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30fc0c2-cff5-4cba-a5c8-3536a3b42d1c admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:08.228942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30fc0c2-cff5-4cba-a5c8-3536a3b42d1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:08.228942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30fc0c2-cff5-4cba-a5c8-3536a3b42d1c admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:08.228942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30fc0c2-cff5-4cba-a5c8-3536a3b42d1c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:56:08.228942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30fc0c2-cff5-4cba-a5c8-3536a3b42d1c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:08.228942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30fc0c2-cff5-4cba-a5c8-3536a3b42d1c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:08.236796 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1244/2490] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:56:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:56:08.241151 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412d64cc-736a-4adf-990e-f24d6b11dab2 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:08.250992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412d64cc-736a-4adf-990e-f24d6b11dab2 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:08.255729 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8cab90-12b0-45f3-97de-bb8d01aa495d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:08.256319 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8cab90-12b0-45f3-97de-bb8d01aa495d admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:08.256595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8cab90-12b0-45f3-97de-bb8d01aa495d admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:08.256861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8cab90-12b0-45f3-97de-bb8d01aa495d admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:08.257538 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8cab90-12b0-45f3-97de-bb8d01aa495d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:08.259520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8cab90-12b0-45f3-97de-bb8d01aa495d admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:08.426474 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1247/2491] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:08 2022] POST /identity/v3/projects => generated 355 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:56:08.430856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30fc0c2-cff5-4cba-a5c8-3536a3b42d1c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:08.440877 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30fc0c2-cff5-4cba-a5c8-3536a3b42d1c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:08.502268 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dce57bf-be9a-4966-92b7-564f860c7b9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:08.502898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dce57bf-be9a-4966-92b7-564f860c7b9b admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:08.503299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dce57bf-be9a-4966-92b7-564f860c7b9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:08.503497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dce57bf-be9a-4966-92b7-564f860c7b9b admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:08.504136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dce57bf-be9a-4966-92b7-564f860c7b9b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:08.506091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dce57bf-be9a-4966-92b7-564f860c7b9b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:08.609419 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:08.615502 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1245/2492] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:08 2022] POST /identity/v3/users => generated 278 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:08.620576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8cab90-12b0-45f3-97de-bb8d01aa495d admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:08.630975 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8cab90-12b0-45f3-97de-bb8d01aa495d admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:08.636384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74a27aa-65a5-4053-ad53-8d79f4f1db52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:08.636978 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74a27aa-65a5-4053-ad53-8d79f4f1db52 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:08.637256 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74a27aa-65a5-4053-ad53-8d79f4f1db52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:08.637525 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74a27aa-65a5-4053-ad53-8d79f4f1db52 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:08.638161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d74a27aa-65a5-4053-ad53-8d79f4f1db52 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:08.639777 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d74a27aa-65a5-4053-ad53-8d79f4f1db52 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:08.651833 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1248/2493] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:08.655901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dce57bf-be9a-4966-92b7-564f860c7b9b admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:08.666642 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dce57bf-be9a-4966-92b7-564f860c7b9b admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:08.671589 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817e267f-8159-442a-ab9b-7fceb22909af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:08.672221 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817e267f-8159-442a-ab9b-7fceb22909af admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:08.672511 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817e267f-8159-442a-ab9b-7fceb22909af admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:08.676011 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817e267f-8159-442a-ab9b-7fceb22909af admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:08.676011 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817e267f-8159-442a-ab9b-7fceb22909af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:08.676011 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817e267f-8159-442a-ab9b-7fceb22909af admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:08.687250 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1246/2494] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:08.691319 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74a27aa-65a5-4053-ad53-8d79f4f1db52 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:08.702290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74a27aa-65a5-4053-ad53-8d79f4f1db52 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:08.707078 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144e40c4-2950-4eee-ae52-4e11d4f27521 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:08.710304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144e40c4-2950-4eee-ae52-4e11d4f27521 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:08.710304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144e40c4-2950-4eee-ae52-4e11d4f27521 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:08.710304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144e40c4-2950-4eee-ae52-4e11d4f27521 admin admin] PATH_INFO: `/v3/projects/f4dbda910bbd48d997a9896c77880f87/users/56c6088d1a0c4e68ad0e2235a8d5981d/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:08.723620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144e40c4-2950-4eee-ae52-4e11d4f27521 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f4dbda910bbd48d997a9896c77880f87, user_id=56c6088d1a0c4e68ad0e2235a8d5981d, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:08.726272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144e40c4-2950-4eee-ae52-4e11d4f27521 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:08.797815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1249/2495] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:08 2022] PUT /identity/v3/projects/f4dbda910bbd48d997a9896c77880f87/users/56c6088d1a0c4e68ad0e2235a8d5981d/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:08.810331 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dab1f7-1a80-4b5c-a736-48c43aa6cc35 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:08.810617 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dab1f7-1a80-4b5c-a736-48c43aa6cc35 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:08.810895 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dab1f7-1a80-4b5c-a736-48c43aa6cc35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:08.840223 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-85dab1f7-1a80-4b5c-a736-48c43aa6cc35 None None] MFA Rules not processed for user `56c6088d1a0c4e68ad0e2235a8d5981d`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:08.841523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-85dab1f7-1a80-4b5c-a736-48c43aa6cc35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:08.916601 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1247/2496] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:08 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:08.921241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144e40c4-2950-4eee-ae52-4e11d4f27521 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:08.984074 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144e40c4-2950-4eee-ae52-4e11d4f27521 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:08.988764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bf9fef-8b35-44db-8559-fca474d0c8b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:08.989360 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bf9fef-8b35-44db-8559-fca474d0c8b4 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:08.989638 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bf9fef-8b35-44db-8559-fca474d0c8b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:08.989935 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bf9fef-8b35-44db-8559-fca474d0c8b4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:08.990550 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bf9fef-8b35-44db-8559-fca474d0c8b4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:08.992615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bf9fef-8b35-44db-8559-fca474d0c8b4 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:09.077937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1250/2497] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:08 2022] POST /identity/v3/projects => generated 357 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:09.082153 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dab1f7-1a80-4b5c-a736-48c43aa6cc35 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:09.093014 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dab1f7-1a80-4b5c-a736-48c43aa6cc35 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:09.150919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39364581-7fec-40e8-acbc-03b055b5cc51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:09.151545 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39364581-7fec-40e8-acbc-03b055b5cc51 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:09.151841 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39364581-7fec-40e8-acbc-03b055b5cc51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:09.152098 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39364581-7fec-40e8-acbc-03b055b5cc51 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:09.152734 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39364581-7fec-40e8-acbc-03b055b5cc51 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:09.154638 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39364581-7fec-40e8-acbc-03b055b5cc51 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:09.295134 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1248/2498] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:09 2022] POST /identity/v3/users => generated 285 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:09.299033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bf9fef-8b35-44db-8559-fca474d0c8b4 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:09.309834 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bf9fef-8b35-44db-8559-fca474d0c8b4 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:09.315446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189295d6-ee17-4c81-82ba-e793175f2afd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:09.315605 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189295d6-ee17-4c81-82ba-e793175f2afd admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:09.315815 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189295d6-ee17-4c81-82ba-e793175f2afd admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:09.316116 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189295d6-ee17-4c81-82ba-e793175f2afd admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:09.316758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189295d6-ee17-4c81-82ba-e793175f2afd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:09.318356 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189295d6-ee17-4c81-82ba-e793175f2afd admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:09.329755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1251/2499] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:09 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:09.334110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39364581-7fec-40e8-acbc-03b055b5cc51 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:09.346129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39364581-7fec-40e8-acbc-03b055b5cc51 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:09.351303 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69397f0c-c8e2-406a-be2c-0ff8b89a58d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:09.352710 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69397f0c-c8e2-406a-be2c-0ff8b89a58d4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:09.352989 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69397f0c-c8e2-406a-be2c-0ff8b89a58d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:09.353262 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69397f0c-c8e2-406a-be2c-0ff8b89a58d4 admin admin] PATH_INFO: `/v3/projects/df4eb1d84244463397cd5c153d7ecc74/users/fb7cc9ae0d4241abba10f65093fa7f06/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:09.368479 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69397f0c-c8e2-406a-be2c-0ff8b89a58d4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df4eb1d84244463397cd5c153d7ecc74, user_id=fb7cc9ae0d4241abba10f65093fa7f06, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:09.371147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69397f0c-c8e2-406a-be2c-0ff8b89a58d4 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:09.425464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1249/2500] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:09 2022] PUT /identity/v3/projects/df4eb1d84244463397cd5c153d7ecc74/users/fb7cc9ae0d4241abba10f65093fa7f06/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:09.430249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189295d6-ee17-4c81-82ba-e793175f2afd admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:09.440834 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189295d6-ee17-4c81-82ba-e793175f2afd admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:09.503580 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0327f19-a55c-4ffd-a361-9ed7c5a2ff84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:09.504196 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0327f19-a55c-4ffd-a361-9ed7c5a2ff84 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:09.504552 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0327f19-a55c-4ffd-a361-9ed7c5a2ff84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:09.504829 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0327f19-a55c-4ffd-a361-9ed7c5a2ff84 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:09.505465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0327f19-a55c-4ffd-a361-9ed7c5a2ff84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:09.507071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0327f19-a55c-4ffd-a361-9ed7c5a2ff84 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:09.520484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1252/2501] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:09 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:09.527351 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69397f0c-c8e2-406a-be2c-0ff8b89a58d4 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:09.540839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69397f0c-c8e2-406a-be2c-0ff8b89a58d4 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:09.545791 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd7730a-f065-447c-9e99-044d586fc329 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:09.546402 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd7730a-f065-447c-9e99-044d586fc329 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:09.546678 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd7730a-f065-447c-9e99-044d586fc329 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:09.546985 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd7730a-f065-447c-9e99-044d586fc329 admin admin] PATH_INFO: `/v3/projects/df4eb1d84244463397cd5c153d7ecc74/users/fb7cc9ae0d4241abba10f65093fa7f06/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:09.548962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd7730a-f065-447c-9e99-044d586fc329 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df4eb1d84244463397cd5c153d7ecc74, user_id=fb7cc9ae0d4241abba10f65093fa7f06, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:09.551858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd7730a-f065-447c-9e99-044d586fc329 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:09.565410 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1250/2502] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:09 2022] PUT /identity/v3/projects/df4eb1d84244463397cd5c153d7ecc74/users/fb7cc9ae0d4241abba10f65093fa7f06/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:09.577575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedbb7e8-3f65-4510-bbd5-83bf9581205d None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:09.577575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedbb7e8-3f65-4510-bbd5-83bf9581205d None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:09.577575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedbb7e8-3f65-4510-bbd5-83bf9581205d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:09.587153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:09.613435 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-eedbb7e8-3f65-4510-bbd5-83bf9581205d None None] MFA Rules not processed for user `fb7cc9ae0d4241abba10f65093fa7f06`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:09.614661 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-eedbb7e8-3f65-4510-bbd5-83bf9581205d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:09.687726 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1253/2503] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:09 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:56:09.700050 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd7730a-f065-447c-9e99-044d586fc329 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:09.701843 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9bd7730a-f065-447c-9e99-044d586fc329 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:09.738488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:09.765653 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd7730a-f065-447c-9e99-044d586fc329 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:09.769560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55159975-69bd-4451-a8ae-62888e5b10ed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:09.772235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55159975-69bd-4451-a8ae-62888e5b10ed service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:09.772235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55159975-69bd-4451-a8ae-62888e5b10ed service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:09.772235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55159975-69bd-4451-a8ae-62888e5b10ed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:09.781228 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55159975-69bd-4451-a8ae-62888e5b10ed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:09.782703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55159975-69bd-4451-a8ae-62888e5b10ed service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:09.793177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1251/2504] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:09 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:09.999847 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eedbb7e8-3f65-4510-bbd5-83bf9581205d None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:10.010126 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eedbb7e8-3f65-4510-bbd5-83bf9581205d None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:10.018511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3de2100-da82-4225-b240-2972c20c94fc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:10.019159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3de2100-da82-4225-b240-2972c20c94fc service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:10.019439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3de2100-da82-4225-b240-2972c20c94fc service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:10.019706 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3de2100-da82-4225-b240-2972c20c94fc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:10.078817 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3de2100-da82-4225-b240-2972c20c94fc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:10.080209 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3de2100-da82-4225-b240-2972c20c94fc service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:10.089832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1254/2505] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:09 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:18.138153 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55159975-69bd-4451-a8ae-62888e5b10ed service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:18.139838 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-55159975-69bd-4451-a8ae-62888e5b10ed service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:18.198827 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55159975-69bd-4451-a8ae-62888e5b10ed service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:18.202486 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4f0fcb-906c-43bc-a2b9-8f5e11337c2b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:18.203138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4f0fcb-906c-43bc-a2b9-8f5e11337c2b service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:18.203417 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4f0fcb-906c-43bc-a2b9-8f5e11337c2b service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:18.203688 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4f0fcb-906c-43bc-a2b9-8f5e11337c2b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:18.205375 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-df4f0fcb-906c-43bc-a2b9-8f5e11337c2b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:18.282867 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4f0fcb-906c-43bc-a2b9-8f5e11337c2b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:18.284405 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4f0fcb-906c-43bc-a2b9-8f5e11337c2b service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:18.302790 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1252/2506] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:19.336399 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3de2100-da82-4225-b240-2972c20c94fc service cinder] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:19.338781 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a3de2100-da82-4225-b240-2972c20c94fc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:19.358244 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3de2100-da82-4225-b240-2972c20c94fc service cinder] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:19.364881 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84bfd2b-f12c-4890-b3e1-ad56af352c7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:19.365670 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84bfd2b-f12c-4890-b3e1-ad56af352c7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:19.366018 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84bfd2b-f12c-4890-b3e1-ad56af352c7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:19.366360 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84bfd2b-f12c-4890-b3e1-ad56af352c7e admin admin] PATH_INFO: `/v3/users/814784d411e9463abeb972279b92e4f8` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:19.367722 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84bfd2b-f12c-4890-b3e1-ad56af352c7e admin admin] RBAC: Authorizing `identity:delete_user(user_id=814784d411e9463abeb972279b92e4f8)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:19.369782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84bfd2b-f12c-4890-b3e1-ad56af352c7e admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:19.616863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a84bfd2b-f12c-4890-b3e1-ad56af352c7e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '814784d411e9463abeb972279b92e4f8'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:19.677562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a84bfd2b-f12c-4890-b3e1-ad56af352c7e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '814784d411e9463abeb972279b92e4f8'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:19.692539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a84bfd2b-f12c-4890-b3e1-ad56af352c7e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '814784d411e9463abeb972279b92e4f8'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:19.693225 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a84bfd2b-f12c-4890-b3e1-ad56af352c7e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '814784d411e9463abeb972279b92e4f8'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:19.701049 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a84bfd2b-f12c-4890-b3e1-ad56af352c7e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '814784d411e9463abeb972279b92e4f8'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:19.703293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:19.707614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1255/2507] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:19 2022] DELETE /identity/v3/users/814784d411e9463abeb972279b92e4f8 => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:19.712319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928e06eb-6b43-4af6-993e-699dc1f0b81c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:19.712858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928e06eb-6b43-4af6-993e-699dc1f0b81c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:19.712962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928e06eb-6b43-4af6-993e-699dc1f0b81c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:19.732179 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-928e06eb-6b43-4af6-993e-699dc1f0b81c None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:19.733475 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-928e06eb-6b43-4af6-993e-699dc1f0b81c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:19.824555 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1253/2508] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:19.835154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84bfd2b-f12c-4890-b3e1-ad56af352c7e admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:19.836802 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a84bfd2b-f12c-4890-b3e1-ad56af352c7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:19.897943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84bfd2b-f12c-4890-b3e1-ad56af352c7e admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:19.912814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa4da2f-9eb4-43cc-ab48-e30ee62921d8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:19.913880 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa4da2f-9eb4-43cc-ab48-e30ee62921d8 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:19.914261 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa4da2f-9eb4-43cc-ab48-e30ee62921d8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:19.914635 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa4da2f-9eb4-43cc-ab48-e30ee62921d8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:19.924608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa4da2f-9eb4-43cc-ab48-e30ee62921d8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:19.926119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa4da2f-9eb4-43cc-ab48-e30ee62921d8 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:19.937467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1256/2509] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:56:20.540226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928e06eb-6b43-4af6-993e-699dc1f0b81c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:20.542951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-928e06eb-6b43-4af6-993e-699dc1f0b81c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:20.560925 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928e06eb-6b43-4af6-993e-699dc1f0b81c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:20.568266 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9a0c77-ba39-4b38-8995-c66c6c5aba47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:20.568936 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9a0c77-ba39-4b38-8995-c66c6c5aba47 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:20.569591 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9a0c77-ba39-4b38-8995-c66c6c5aba47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:20.569894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9a0c77-ba39-4b38-8995-c66c6c5aba47 admin admin] PATH_INFO: `/v3/projects/db87e234bcbb4da7923ff645da7ceed5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:20.571168 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9a0c77-ba39-4b38-8995-c66c6c5aba47 admin admin] RBAC: Authorizing `identity:delete_project(project_id=db87e234bcbb4da7923ff645da7ceed5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:20.575226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9a0c77-ba39-4b38-8995-c66c6c5aba47 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:20.607765 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:20.695885 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-dd9a0c77-ba39-4b38-8995-c66c6c5aba47 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db87e234bcbb4da7923ff645da7ceed5', 'request_id': 'req-dd9a0c77-ba39-4b38-8995-c66c6c5aba47'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:20.704236 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-dd9a0c77-ba39-4b38-8995-c66c6c5aba47 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db87e234bcbb4da7923ff645da7ceed5', 'request_id': 'req-dd9a0c77-ba39-4b38-8995-c66c6c5aba47'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:20.773464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-dd9a0c77-ba39-4b38-8995-c66c6c5aba47 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db87e234bcbb4da7923ff645da7ceed5', 'request_id': 'req-dd9a0c77-ba39-4b38-8995-c66c6c5aba47'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:20.785262 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-dd9a0c77-ba39-4b38-8995-c66c6c5aba47 admin admin] The token cache is being invalidate because project db87e234bcbb4da7923ff645da7ceed5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:20.785663 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-dd9a0c77-ba39-4b38-8995-c66c6c5aba47 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:20.786374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-dd9a0c77-ba39-4b38-8995-c66c6c5aba47 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:20.788222 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1254/2510] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:20 2022] DELETE /identity/v3/projects/db87e234bcbb4da7923ff645da7ceed5 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:20.792610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa4da2f-9eb4-43cc-ab48-e30ee62921d8 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:20.802812 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa4da2f-9eb4-43cc-ab48-e30ee62921d8 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:20.864566 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5956ec-c587-4bd3-b830-193b4632887b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:20.865133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5956ec-c587-4bd3-b830-193b4632887b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:20.865409 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5956ec-c587-4bd3-b830-193b4632887b admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:20.865679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5956ec-c587-4bd3-b830-193b4632887b admin admin] PATH_INFO: `/v3/users/9b291a82a59945fbb5c82e1925798157` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:20.866790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5956ec-c587-4bd3-b830-193b4632887b admin admin] RBAC: Authorizing `identity:delete_user(user_id=9b291a82a59945fbb5c82e1925798157)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:20.869575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5956ec-c587-4bd3-b830-193b4632887b admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:21.073955 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6b5956ec-c587-4bd3-b830-193b4632887b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9b291a82a59945fbb5c82e1925798157'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:21.144868 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6b5956ec-c587-4bd3-b830-193b4632887b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9b291a82a59945fbb5c82e1925798157'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:21.160893 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6b5956ec-c587-4bd3-b830-193b4632887b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9b291a82a59945fbb5c82e1925798157'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:21.161982 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6b5956ec-c587-4bd3-b830-193b4632887b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9b291a82a59945fbb5c82e1925798157'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:21.174629 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6b5956ec-c587-4bd3-b830-193b4632887b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9b291a82a59945fbb5c82e1925798157'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:21.185691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1257/2511] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:20 2022] DELETE /identity/v3/users/9b291a82a59945fbb5c82e1925798157 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:21.729442 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9a0c77-ba39-4b38-8995-c66c6c5aba47 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:21.732624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-dd9a0c77-ba39-4b38-8995-c66c6c5aba47 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:21.757890 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9a0c77-ba39-4b38-8995-c66c6c5aba47 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:21.838507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b45b67-d815-4f95-9ddd-848dc0ff3f89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:21.839274 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b45b67-d815-4f95-9ddd-848dc0ff3f89 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:21.839448 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b45b67-d815-4f95-9ddd-848dc0ff3f89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:21.839939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b45b67-d815-4f95-9ddd-848dc0ff3f89 admin admin] PATH_INFO: `/v3/projects/ccc66f18155d46c1ba4dc93f4f69a28f` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:21.841371 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b45b67-d815-4f95-9ddd-848dc0ff3f89 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ccc66f18155d46c1ba4dc93f4f69a28f)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:21.843373 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b45b67-d815-4f95-9ddd-848dc0ff3f89 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:21.988079 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c1b45b67-d815-4f95-9ddd-848dc0ff3f89 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ccc66f18155d46c1ba4dc93f4f69a28f', 'request_id': 'req-c1b45b67-d815-4f95-9ddd-848dc0ff3f89'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:21.996389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c1b45b67-d815-4f95-9ddd-848dc0ff3f89 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ccc66f18155d46c1ba4dc93f4f69a28f', 'request_id': 'req-c1b45b67-d815-4f95-9ddd-848dc0ff3f89'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:22.087946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c1b45b67-d815-4f95-9ddd-848dc0ff3f89 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ccc66f18155d46c1ba4dc93f4f69a28f', 'request_id': 'req-c1b45b67-d815-4f95-9ddd-848dc0ff3f89'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:22.100745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c1b45b67-d815-4f95-9ddd-848dc0ff3f89 admin admin] The token cache is being invalidate because project ccc66f18155d46c1ba4dc93f4f69a28f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:22.101040 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c1b45b67-d815-4f95-9ddd-848dc0ff3f89 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:22.101695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c1b45b67-d815-4f95-9ddd-848dc0ff3f89 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:22.103266 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1255/2512] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:21 2022] DELETE /identity/v3/projects/ccc66f18155d46c1ba4dc93f4f69a28f => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:22.109382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4519025c-89c7-4be1-8760-ecd884af57b0 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:22.109811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4519025c-89c7-4be1-8760-ecd884af57b0 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:22.110085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4519025c-89c7-4be1-8760-ecd884af57b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:22.131830 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-4519025c-89c7-4be1-8760-ecd884af57b0 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:22.133182 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-4519025c-89c7-4be1-8760-ecd884af57b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:22.152228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:22.198680 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1258/2513] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:22.208016 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa88c417-cb1b-4cbe-9d13-0cff0738b6cc None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:22.208297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa88c417-cb1b-4cbe-9d13-0cff0738b6cc None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:22.208568 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa88c417-cb1b-4cbe-9d13-0cff0738b6cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:22.230983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-aa88c417-cb1b-4cbe-9d13-0cff0738b6cc None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:22.232209 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-aa88c417-cb1b-4cbe-9d13-0cff0738b6cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:22.240146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1256/2514] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:22.245046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4519025c-89c7-4be1-8760-ecd884af57b0 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:22.255290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4519025c-89c7-4be1-8760-ecd884af57b0 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:22.262588 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59081110-23df-409d-8ad7-27c4f4b09468 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:22.263318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59081110-23df-409d-8ad7-27c4f4b09468 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:22.263669 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59081110-23df-409d-8ad7-27c4f4b09468 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:22.264014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59081110-23df-409d-8ad7-27c4f4b09468 admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:22.264740 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59081110-23df-409d-8ad7-27c4f4b09468 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:56:22.265113 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59081110-23df-409d-8ad7-27c4f4b09468 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:22.266838 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59081110-23df-409d-8ad7-27c4f4b09468 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:22.279908 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1259/2515] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:56:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:56:22.284271 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa88c417-cb1b-4cbe-9d13-0cff0738b6cc None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:22.294960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa88c417-cb1b-4cbe-9d13-0cff0738b6cc None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:22.300647 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e0950e-44ff-4e74-8e5e-3657a8dc87f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:22.301318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e0950e-44ff-4e74-8e5e-3657a8dc87f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:22.301603 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e0950e-44ff-4e74-8e5e-3657a8dc87f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:22.301997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e0950e-44ff-4e74-8e5e-3657a8dc87f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:22.302638 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e0950e-44ff-4e74-8e5e-3657a8dc87f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:22.304797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e0950e-44ff-4e74-8e5e-3657a8dc87f5 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:22.525016 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1257/2516] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:22 2022] POST /identity/v3/projects => generated 383 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:22.526273 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59081110-23df-409d-8ad7-27c4f4b09468 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:22.528807 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-59081110-23df-409d-8ad7-27c4f4b09468 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:22.537843 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e0950e-44ff-4e74-8e5e-3657a8dc87f5 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:22.545131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59081110-23df-409d-8ad7-27c4f4b09468 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:22.552071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:22.609828 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fa6ee2-daf3-4770-b7cb-a478a4ee9629 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:22.610607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fa6ee2-daf3-4770-b7cb-a478a4ee9629 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:22.610895 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fa6ee2-daf3-4770-b7cb-a478a4ee9629 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:22.611191 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fa6ee2-daf3-4770-b7cb-a478a4ee9629 admin admin] PATH_INFO: `/v3/users/416479ba056f4612b6214cf115251269` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:22.612464 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fa6ee2-daf3-4770-b7cb-a478a4ee9629 admin admin] RBAC: Authorizing `identity:delete_user(user_id=416479ba056f4612b6214cf115251269)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:22.612736 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e0950e-44ff-4e74-8e5e-3657a8dc87f5 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:22.614401 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fa6ee2-daf3-4770-b7cb-a478a4ee9629 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:22.617696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba12adac-ea82-4aa9-9656-ae34bd967114 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:22.618348 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba12adac-ea82-4aa9-9656-ae34bd967114 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:22.618690 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba12adac-ea82-4aa9-9656-ae34bd967114 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:22.619858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba12adac-ea82-4aa9-9656-ae34bd967114 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:22.620176 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba12adac-ea82-4aa9-9656-ae34bd967114 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:22.622093 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba12adac-ea82-4aa9-9656-ae34bd967114 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:22.740466 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1258/2517] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:22 2022] POST /identity/v3/users => generated 292 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:22.744705 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba12adac-ea82-4aa9-9656-ae34bd967114 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:22.755511 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba12adac-ea82-4aa9-9656-ae34bd967114 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:22.760838 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8056b7a2-2002-4c40-8f52-c6ee05d58168 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:22.761540 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8056b7a2-2002-4c40-8f52-c6ee05d58168 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:22.761893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8056b7a2-2002-4c40-8f52-c6ee05d58168 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:22.762140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8056b7a2-2002-4c40-8f52-c6ee05d58168 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:22.762828 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8056b7a2-2002-4c40-8f52-c6ee05d58168 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:22.764422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8056b7a2-2002-4c40-8f52-c6ee05d58168 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:22.777160 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1259/2518] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:22 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:56:22.777160 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8056b7a2-2002-4c40-8f52-c6ee05d58168 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:22.779039 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8056b7a2-2002-4c40-8f52-c6ee05d58168 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:22.857354 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-85fa6ee2-daf3-4770-b7cb-a478a4ee9629 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '416479ba056f4612b6214cf115251269'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:22.872604 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8056b7a2-2002-4c40-8f52-c6ee05d58168 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:22.879625 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5f16e5-8c46-4912-abf0-712dc12651ac service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '176da4679fcb46148da9289fccdad5c2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:22.881565 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5f16e5-8c46-4912-abf0-712dc12651ac service placement] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:22.881565 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5f16e5-8c46-4912-abf0-712dc12651ac service placement] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:22.881565 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5f16e5-8c46-4912-abf0-712dc12651ac service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:22.882754 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ae5f16e5-8c46-4912-abf0-712dc12651ac service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:22.894822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5f16e5-8c46-4912-abf0-712dc12651ac service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:22.896442 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5f16e5-8c46-4912-abf0-712dc12651ac service placement] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:22.907198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1260/2519] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:22 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:22.907990 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5f16e5-8c46-4912-abf0-712dc12651ac service placement] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:22.918657 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5f16e5-8c46-4912-abf0-712dc12651ac service placement] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:22.923353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74f0bc2-403a-4af6-8c4e-be3214c696db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:22.923929 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74f0bc2-403a-4af6-8c4e-be3214c696db admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:22.924207 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74f0bc2-403a-4af6-8c4e-be3214c696db admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:22.924477 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74f0bc2-403a-4af6-8c4e-be3214c696db admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:22.925093 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74f0bc2-403a-4af6-8c4e-be3214c696db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:22.926706 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74f0bc2-403a-4af6-8c4e-be3214c696db admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:22.937476 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1261/2520] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:22.941254 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-85fa6ee2-daf3-4770-b7cb-a478a4ee9629 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '416479ba056f4612b6214cf115251269'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:22.941952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74f0bc2-403a-4af6-8c4e-be3214c696db admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:22.954042 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74f0bc2-403a-4af6-8c4e-be3214c696db admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:22.956615 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-85fa6ee2-daf3-4770-b7cb-a478a4ee9629 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '416479ba056f4612b6214cf115251269'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:22.957467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-85fa6ee2-daf3-4770-b7cb-a478a4ee9629 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '416479ba056f4612b6214cf115251269'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:22.959272 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2ca117-a2fd-4501-aad5-ed8a3093c9ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:22.959926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2ca117-a2fd-4501-aad5-ed8a3093c9ea admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:22.960333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2ca117-a2fd-4501-aad5-ed8a3093c9ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:22.960612 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2ca117-a2fd-4501-aad5-ed8a3093c9ea admin admin] PATH_INFO: `/v3/projects/941f0e8d4caa4f0a8d8289cfdb34d435/users/fd4fdaf3846949709314ef3b230fcf1e/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:22.965988 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-85fa6ee2-daf3-4770-b7cb-a478a4ee9629 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '416479ba056f4612b6214cf115251269'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:22.972305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1260/2521] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:22 2022] DELETE /identity/v3/users/416479ba056f4612b6214cf115251269 => generated 0 bytes in 447 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:22.976560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2ca117-a2fd-4501-aad5-ed8a3093c9ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=941f0e8d4caa4f0a8d8289cfdb34d435, user_id=fd4fdaf3846949709314ef3b230fcf1e, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:22.979765 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2ca117-a2fd-4501-aad5-ed8a3093c9ea admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:23.076560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1262/2522] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:22 2022] PUT /identity/v3/projects/941f0e8d4caa4f0a8d8289cfdb34d435/users/fd4fdaf3846949709314ef3b230fcf1e/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:23.097117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247facd4-af86-46ff-ad13-3ff9c43e6779 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:23.104184 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247facd4-af86-46ff-ad13-3ff9c43e6779 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:23.104477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247facd4-af86-46ff-ad13-3ff9c43e6779 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:23.170195 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-247facd4-af86-46ff-ad13-3ff9c43e6779 None None] MFA Rules not processed for user `fd4fdaf3846949709314ef3b230fcf1e`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:23.171648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-247facd4-af86-46ff-ad13-3ff9c43e6779 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:23.190755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:23.244676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1261/2523] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:23 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:23.249160 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2ca117-a2fd-4501-aad5-ed8a3093c9ea admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:23.251120 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-dd2ca117-a2fd-4501-aad5-ed8a3093c9ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:23.276806 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247facd4-af86-46ff-ad13-3ff9c43e6779 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:23.278811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-247facd4-af86-46ff-ad13-3ff9c43e6779 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:23.284372 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:23.338437 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2ca117-a2fd-4501-aad5-ed8a3093c9ea admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:23.343356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65913354-7edc-464c-ad03-cbd9afa555d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:23.343970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65913354-7edc-464c-ad03-cbd9afa555d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:23.344253 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65913354-7edc-464c-ad03-cbd9afa555d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:23.344534 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65913354-7edc-464c-ad03-cbd9afa555d8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:23.345166 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65913354-7edc-464c-ad03-cbd9afa555d8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:23.347086 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65913354-7edc-464c-ad03-cbd9afa555d8 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:23.376315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247facd4-af86-46ff-ad13-3ff9c43e6779 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:23.381089 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505ae25c-1592-4de6-b349-15384f2c52bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:23.381751 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505ae25c-1592-4de6-b349-15384f2c52bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:23.382098 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505ae25c-1592-4de6-b349-15384f2c52bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:23.382383 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505ae25c-1592-4de6-b349-15384f2c52bc admin admin] PATH_INFO: `/v3/projects/cb698ddcedae42a29f50203ec59db295` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:23.383540 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505ae25c-1592-4de6-b349-15384f2c52bc admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb698ddcedae42a29f50203ec59db295)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:23.386173 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505ae25c-1592-4de6-b349-15384f2c52bc admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:23.489502 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1263/2524] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:23 2022] POST /identity/v3/projects => generated 379 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:23.493963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65913354-7edc-464c-ad03-cbd9afa555d8 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:23.523426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-505ae25c-1592-4de6-b349-15384f2c52bc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb698ddcedae42a29f50203ec59db295', 'request_id': 'req-505ae25c-1592-4de6-b349-15384f2c52bc'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:23.555344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65913354-7edc-464c-ad03-cbd9afa555d8 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:23.613241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86afc483-ba02-4376-b9a1-317d5030602b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:23.613898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86afc483-ba02-4376-b9a1-317d5030602b admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:23.614179 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86afc483-ba02-4376-b9a1-317d5030602b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:23.614457 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86afc483-ba02-4376-b9a1-317d5030602b admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:23.615197 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86afc483-ba02-4376-b9a1-317d5030602b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:23.617176 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86afc483-ba02-4376-b9a1-317d5030602b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:23.687999 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-505ae25c-1592-4de6-b349-15384f2c52bc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb698ddcedae42a29f50203ec59db295', 'request_id': 'req-505ae25c-1592-4de6-b349-15384f2c52bc'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:23.694065 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-505ae25c-1592-4de6-b349-15384f2c52bc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb698ddcedae42a29f50203ec59db295', 'request_id': 'req-505ae25c-1592-4de6-b349-15384f2c52bc'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:23.706461 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-505ae25c-1592-4de6-b349-15384f2c52bc admin admin] The token cache is being invalidate because project cb698ddcedae42a29f50203ec59db295 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:23.706752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-505ae25c-1592-4de6-b349-15384f2c52bc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:23.707425 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-505ae25c-1592-4de6-b349-15384f2c52bc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:23.708937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1262/2525] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:23 2022] DELETE /identity/v3/projects/cb698ddcedae42a29f50203ec59db295 => generated 0 bytes in 433 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:56:23.713741 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727fbfa7-f63f-48df-8c5c-27db5c644ec1 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:23.714031 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727fbfa7-f63f-48df-8c5c-27db5c644ec1 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:23.714315 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727fbfa7-f63f-48df-8c5c-27db5c644ec1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:23.734896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1264/2526] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:23 2022] POST /identity/v3/users => generated 296 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:56:23.737201 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-727fbfa7-f63f-48df-8c5c-27db5c644ec1 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:23.738801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86afc483-ba02-4376-b9a1-317d5030602b admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:23.739002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-727fbfa7-f63f-48df-8c5c-27db5c644ec1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:23.741119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-86afc483-ba02-4376-b9a1-317d5030602b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:23.747616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1263/2527] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:56:23.754314 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86afc483-ba02-4376-b9a1-317d5030602b admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:23.755636 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bd8628-a0f6-4b31-bd6f-d0fff5f50847 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:23.756551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bd8628-a0f6-4b31-bd6f-d0fff5f50847 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:23.756551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bd8628-a0f6-4b31-bd6f-d0fff5f50847 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:23.759149 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6021768-3398-4ce2-b4e0-5661829a2a03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:23.759758 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6021768-3398-4ce2-b4e0-5661829a2a03 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:23.760040 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6021768-3398-4ce2-b4e0-5661829a2a03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:23.760317 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6021768-3398-4ce2-b4e0-5661829a2a03 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:23.760991 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6021768-3398-4ce2-b4e0-5661829a2a03 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:23.762570 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6021768-3398-4ce2-b4e0-5661829a2a03 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:23.772979 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1265/2528] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:23 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:56:23.777643 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6021768-3398-4ce2-b4e0-5661829a2a03 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:23.778318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-f0bd8628-a0f6-4b31-bd6f-d0fff5f50847 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:23.779761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-f0bd8628-a0f6-4b31-bd6f-d0fff5f50847 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:23.788005 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1264/2529] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:56:23.797185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6021768-3398-4ce2-b4e0-5661829a2a03 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:23.797185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994da5c0-8700-41dc-90af-400ff7e525b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:23.797185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994da5c0-8700-41dc-90af-400ff7e525b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:23.797185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994da5c0-8700-41dc-90af-400ff7e525b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:23.797596 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994da5c0-8700-41dc-90af-400ff7e525b9 admin admin] PATH_INFO: `/v3/projects/d564dfea143441c496cead0c412288c9/users/e63897f83ea24e8e93ad52f4a68cb3bc/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:23.800050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bd8628-a0f6-4b31-bd6f-d0fff5f50847 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:23.810556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bd8628-a0f6-4b31-bd6f-d0fff5f50847 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:23.812622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994da5c0-8700-41dc-90af-400ff7e525b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d564dfea143441c496cead0c412288c9, user_id=e63897f83ea24e8e93ad52f4a68cb3bc, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:23.815336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994da5c0-8700-41dc-90af-400ff7e525b9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:23.815478 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95d72d2-20ee-41d5-ae3c-25c5a6c97b2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:23.816048 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95d72d2-20ee-41d5-ae3c-25c5a6c97b2a admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:23.816325 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95d72d2-20ee-41d5-ae3c-25c5a6c97b2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:23.816604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95d72d2-20ee-41d5-ae3c-25c5a6c97b2a admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:23.817240 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95d72d2-20ee-41d5-ae3c-25c5a6c97b2a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:56:23.817544 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95d72d2-20ee-41d5-ae3c-25c5a6c97b2a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:23.819142 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95d72d2-20ee-41d5-ae3c-25c5a6c97b2a admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:23.835215 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1265/2530] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:56:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:56:23.836937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95d72d2-20ee-41d5-ae3c-25c5a6c97b2a admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:23.849041 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95d72d2-20ee-41d5-ae3c-25c5a6c97b2a admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:23.855316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6edee52-cc1a-4895-bcec-e870718c0cc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:23.855998 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6edee52-cc1a-4895-bcec-e870718c0cc9 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:23.856448 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6edee52-cc1a-4895-bcec-e870718c0cc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:23.856800 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6edee52-cc1a-4895-bcec-e870718c0cc9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:23.857437 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6edee52-cc1a-4895-bcec-e870718c0cc9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:23.859433 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6edee52-cc1a-4895-bcec-e870718c0cc9 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:23.889511 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1266/2531] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:23 2022] PUT /identity/v3/projects/d564dfea143441c496cead0c412288c9/users/e63897f83ea24e8e93ad52f4a68cb3bc/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:56:23.893495 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994da5c0-8700-41dc-90af-400ff7e525b9 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:23.960842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1266/2532] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:23 2022] POST /identity/v3/projects => generated 383 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:56:23.966155 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6edee52-cc1a-4895-bcec-e870718c0cc9 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:23.979415 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994da5c0-8700-41dc-90af-400ff7e525b9 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:24.035381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6edee52-cc1a-4895-bcec-e870718c0cc9 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:24.040265 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecae939-a37b-4b50-879b-e6461c60cc8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:24.040409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd6b7ff-5766-4fa8-83e8-6667def79fbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:24.041476 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd6b7ff-5766-4fa8-83e8-6667def79fbb admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:24.041755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecae939-a37b-4b50-879b-e6461c60cc8b admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:24.041880 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd6b7ff-5766-4fa8-83e8-6667def79fbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:24.042128 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd6b7ff-5766-4fa8-83e8-6667def79fbb admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:24.042248 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecae939-a37b-4b50-879b-e6461c60cc8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:24.042514 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecae939-a37b-4b50-879b-e6461c60cc8b admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:24.042760 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd6b7ff-5766-4fa8-83e8-6667def79fbb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:24.043331 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ecae939-a37b-4b50-879b-e6461c60cc8b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:24.044459 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd6b7ff-5766-4fa8-83e8-6667def79fbb admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:24.045380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ecae939-a37b-4b50-879b-e6461c60cc8b admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:24.054227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1267/2533] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:23 2022] GET /identity/v3/roles => generated 1420 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:56:24.059288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd6b7ff-5766-4fa8-83e8-6667def79fbb admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:24.068835 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd6b7ff-5766-4fa8-83e8-6667def79fbb admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:24.073500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d561fa-d66a-4708-93c2-ce638a48b125 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:24.074143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d561fa-d66a-4708-93c2-ce638a48b125 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:24.074421 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d561fa-d66a-4708-93c2-ce638a48b125 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:24.074690 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d561fa-d66a-4708-93c2-ce638a48b125 admin admin] PATH_INFO: `/v3/projects/d564dfea143441c496cead0c412288c9/users/e63897f83ea24e8e93ad52f4a68cb3bc/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:24.076695 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d561fa-d66a-4708-93c2-ce638a48b125 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d564dfea143441c496cead0c412288c9, user_id=e63897f83ea24e8e93ad52f4a68cb3bc, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:24.079348 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d561fa-d66a-4708-93c2-ce638a48b125 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:24.090186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1268/2534] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:24 2022] PUT /identity/v3/projects/d564dfea143441c496cead0c412288c9/users/e63897f83ea24e8e93ad52f4a68cb3bc/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:56:24.100765 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5979eb7f-4de5-43eb-8da4-eaaa34f0483a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:24.101048 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5979eb7f-4de5-43eb-8da4-eaaa34f0483a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:24.101318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5979eb7f-4de5-43eb-8da4-eaaa34f0483a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:24.117287 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:24.131480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-5979eb7f-4de5-43eb-8da4-eaaa34f0483a None None] MFA Rules not processed for user `e63897f83ea24e8e93ad52f4a68cb3bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:24.132775 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5979eb7f-4de5-43eb-8da4-eaaa34f0483a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:24.166102 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1267/2535] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:23 2022] POST /identity/v3/users => generated 292 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:56:24.167277 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecae939-a37b-4b50-879b-e6461c60cc8b admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:24.177272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecae939-a37b-4b50-879b-e6461c60cc8b admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:24.187813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:24.200981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1269/2536] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:24 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:56:24.211861 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5979eb7f-4de5-43eb-8da4-eaaa34f0483a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:24.213972 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5979eb7f-4de5-43eb-8da4-eaaa34f0483a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:24.237776 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e31203b-a3b8-4bcd-a315-115ec66fff1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:24.238431 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e31203b-a3b8-4bcd-a315-115ec66fff1e admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:24.238719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e31203b-a3b8-4bcd-a315-115ec66fff1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:24.239033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e31203b-a3b8-4bcd-a315-115ec66fff1e admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:24.239678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e31203b-a3b8-4bcd-a315-115ec66fff1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:24.241344 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e31203b-a3b8-4bcd-a315-115ec66fff1e admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:24.251488 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1268/2537] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:24 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:56:24.255710 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e31203b-a3b8-4bcd-a315-115ec66fff1e admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:24.266698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e31203b-a3b8-4bcd-a315-115ec66fff1e admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:24.270387 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5979eb7f-4de5-43eb-8da4-eaaa34f0483a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:24.272829 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06b6e8b-5ce6-48fc-9c51-71cb09081742 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:24.273870 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06b6e8b-5ce6-48fc-9c51-71cb09081742 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:24.274342 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b943c8c-4b21-453c-8305-b67d6d4b606b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:24.274462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06b6e8b-5ce6-48fc-9c51-71cb09081742 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:24.274809 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06b6e8b-5ce6-48fc-9c51-71cb09081742 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:24.275854 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b943c8c-4b21-453c-8305-b67d6d4b606b service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:24.276270 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b943c8c-4b21-453c-8305-b67d6d4b606b service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:24.276510 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06b6e8b-5ce6-48fc-9c51-71cb09081742 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:24.276693 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b943c8c-4b21-453c-8305-b67d6d4b606b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:24.279523 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06b6e8b-5ce6-48fc-9c51-71cb09081742 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:24.288118 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b943c8c-4b21-453c-8305-b67d6d4b606b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:24.293162 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b943c8c-4b21-453c-8305-b67d6d4b606b service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:24.293318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1269/2538] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:24 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:56:24.297266 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06b6e8b-5ce6-48fc-9c51-71cb09081742 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:24.301674 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1270/2539] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:24 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:56:24.307605 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06b6e8b-5ce6-48fc-9c51-71cb09081742 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:24.312916 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1c18cc-605f-4087-9820-9ad4093b9431 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:24.313582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1c18cc-605f-4087-9820-9ad4093b9431 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:24.314032 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1c18cc-605f-4087-9820-9ad4093b9431 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:24.314317 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1c18cc-605f-4087-9820-9ad4093b9431 admin admin] PATH_INFO: `/v3/projects/fb7f1e80fe074809bc81d3b9c98d0e2e/users/cbae7d1df881469f97348d037e3d2ec2/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:24.337125 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1c18cc-605f-4087-9820-9ad4093b9431 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb7f1e80fe074809bc81d3b9c98d0e2e, user_id=cbae7d1df881469f97348d037e3d2ec2, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:24.345961 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1c18cc-605f-4087-9820-9ad4093b9431 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:24.414119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1270/2540] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:24 2022] PUT /identity/v3/projects/fb7f1e80fe074809bc81d3b9c98d0e2e/users/cbae7d1df881469f97348d037e3d2ec2/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:24.422557 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b943c8c-4b21-453c-8305-b67d6d4b606b service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:24.426251 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be9d1e9-1b03-4fee-8ec8-2661729d9a7c None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:24.426613 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be9d1e9-1b03-4fee-8ec8-2661729d9a7c None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:24.426891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be9d1e9-1b03-4fee-8ec8-2661729d9a7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:24.431575 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b943c8c-4b21-453c-8305-b67d6d4b606b service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:24.434209 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:24.455384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-3be9d1e9-1b03-4fee-8ec8-2661729d9a7c None None] MFA Rules not processed for user `cbae7d1df881469f97348d037e3d2ec2`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:24.456502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3be9d1e9-1b03-4fee-8ec8-2661729d9a7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:24.485812 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84cc33b-8e33-428c-a204-df5ec2d0d68d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:24.486626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84cc33b-8e33-428c-a204-df5ec2d0d68d service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:24.486926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84cc33b-8e33-428c-a204-df5ec2d0d68d service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:24.487203 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84cc33b-8e33-428c-a204-df5ec2d0d68d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:24.488950 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b84cc33b-8e33-428c-a204-df5ec2d0d68d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:24.523959 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1271/2541] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:24 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:24.536043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be9d1e9-1b03-4fee-8ec8-2661729d9a7c None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:24.537473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3be9d1e9-1b03-4fee-8ec8-2661729d9a7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:24.555983 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84cc33b-8e33-428c-a204-df5ec2d0d68d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:24.558016 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84cc33b-8e33-428c-a204-df5ec2d0d68d service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:24.567693 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1271/2542] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:24 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:24.649717 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be9d1e9-1b03-4fee-8ec8-2661729d9a7c None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:24.657806 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6c1e86-4e06-4f08-a62b-36b60cd9e5a0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:24.658674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6c1e86-4e06-4f08-a62b-36b60cd9e5a0 service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:24.659148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6c1e86-4e06-4f08-a62b-36b60cd9e5a0 service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:24.659553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6c1e86-4e06-4f08-a62b-36b60cd9e5a0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:24.672524 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6c1e86-4e06-4f08-a62b-36b60cd9e5a0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:24.674149 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6c1e86-4e06-4f08-a62b-36b60cd9e5a0 service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:24.680630 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:24.687760 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1272/2543] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:24 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:28.030488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84cc33b-8e33-428c-a204-df5ec2d0d68d service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:28.096184 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84cc33b-8e33-428c-a204-df5ec2d0d68d service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:28.100686 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bc8467-6b06-46bf-8e71-6a63fca2c3d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:28.101306 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bc8467-6b06-46bf-8e71-6a63fca2c3d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:28.101584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bc8467-6b06-46bf-8e71-6a63fca2c3d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:28.101856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bc8467-6b06-46bf-8e71-6a63fca2c3d4 admin admin] PATH_INFO: `/v3/users/fd4fdaf3846949709314ef3b230fcf1e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:28.102975 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27bc8467-6b06-46bf-8e71-6a63fca2c3d4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fd4fdaf3846949709314ef3b230fcf1e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:28.105430 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27bc8467-6b06-46bf-8e71-6a63fca2c3d4 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:28.127480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:28.348875 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-27bc8467-6b06-46bf-8e71-6a63fca2c3d4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fd4fdaf3846949709314ef3b230fcf1e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:28.362797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-27bc8467-6b06-46bf-8e71-6a63fca2c3d4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fd4fdaf3846949709314ef3b230fcf1e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:28.436681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-27bc8467-6b06-46bf-8e71-6a63fca2c3d4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fd4fdaf3846949709314ef3b230fcf1e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:28.437395 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-27bc8467-6b06-46bf-8e71-6a63fca2c3d4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fd4fdaf3846949709314ef3b230fcf1e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:28.445338 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-27bc8467-6b06-46bf-8e71-6a63fca2c3d4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fd4fdaf3846949709314ef3b230fcf1e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:28.452436 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1272/2544] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:28 2022] DELETE /identity/v3/users/fd4fdaf3846949709314ef3b230fcf1e => generated 0 bytes in 422 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:28.456700 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2049be-7fc4-454c-b832-6a026ee1d6ae None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:28.457187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2049be-7fc4-454c-b832-6a026ee1d6ae None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:28.457296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2049be-7fc4-454c-b832-6a026ee1d6ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:28.479995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-df2049be-7fc4-454c-b832-6a026ee1d6ae None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:28.482734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-df2049be-7fc4-454c-b832-6a026ee1d6ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:28.553605 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1273/2545] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:28.566932 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bc8467-6b06-46bf-8e71-6a63fca2c3d4 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:28.568411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-27bc8467-6b06-46bf-8e71-6a63fca2c3d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:28.631096 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bc8467-6b06-46bf-8e71-6a63fca2c3d4 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:28.634795 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb47caea-8a60-4d14-ba49-0147f8ce18c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:28.635426 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47caea-8a60-4d14-ba49-0147f8ce18c3 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:28.635779 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47caea-8a60-4d14-ba49-0147f8ce18c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:28.636772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47caea-8a60-4d14-ba49-0147f8ce18c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:28.646555 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb47caea-8a60-4d14-ba49-0147f8ce18c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:28.648045 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb47caea-8a60-4d14-ba49-0147f8ce18c3 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:28.657963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1273/2546] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:29.123752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2049be-7fc4-454c-b832-6a026ee1d6ae None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:29.125246 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-df2049be-7fc4-454c-b832-6a026ee1d6ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:29.140442 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2049be-7fc4-454c-b832-6a026ee1d6ae None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:29.145064 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aff961c-ebd8-4a40-8856-2d435620131b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:29.145705 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aff961c-ebd8-4a40-8856-2d435620131b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:29.145987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aff961c-ebd8-4a40-8856-2d435620131b admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:29.146262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aff961c-ebd8-4a40-8856-2d435620131b admin admin] PATH_INFO: `/v3/projects/941f0e8d4caa4f0a8d8289cfdb34d435` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:29.147532 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aff961c-ebd8-4a40-8856-2d435620131b admin admin] RBAC: Authorizing `identity:delete_project(project_id=941f0e8d4caa4f0a8d8289cfdb34d435)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:29.149503 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aff961c-ebd8-4a40-8856-2d435620131b admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:29.285625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:29.294736 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-8aff961c-ebd8-4a40-8856-2d435620131b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '941f0e8d4caa4f0a8d8289cfdb34d435', 'request_id': 'req-8aff961c-ebd8-4a40-8856-2d435620131b'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:29.390267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-8aff961c-ebd8-4a40-8856-2d435620131b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '941f0e8d4caa4f0a8d8289cfdb34d435', 'request_id': 'req-8aff961c-ebd8-4a40-8856-2d435620131b'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:29.396574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-8aff961c-ebd8-4a40-8856-2d435620131b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '941f0e8d4caa4f0a8d8289cfdb34d435', 'request_id': 'req-8aff961c-ebd8-4a40-8856-2d435620131b'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:29.408634 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-8aff961c-ebd8-4a40-8856-2d435620131b admin admin] The token cache is being invalidate because project 941f0e8d4caa4f0a8d8289cfdb34d435 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:29.408924 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-8aff961c-ebd8-4a40-8856-2d435620131b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:29.409561 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-8aff961c-ebd8-4a40-8856-2d435620131b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:29.411121 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1274/2547] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:29 2022] DELETE /identity/v3/projects/941f0e8d4caa4f0a8d8289cfdb34d435 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:29.415131 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb47caea-8a60-4d14-ba49-0147f8ce18c3 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:29.425504 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb47caea-8a60-4d14-ba49-0147f8ce18c3 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:29.484425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0200acc-b57c-4276-998f-27d1cc778b7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:29.485061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0200acc-b57c-4276-998f-27d1cc778b7b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:29.485339 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0200acc-b57c-4276-998f-27d1cc778b7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:29.485613 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0200acc-b57c-4276-998f-27d1cc778b7b admin admin] PATH_INFO: `/v3/users/e63897f83ea24e8e93ad52f4a68cb3bc` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:29.486773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0200acc-b57c-4276-998f-27d1cc778b7b admin admin] RBAC: Authorizing `identity:delete_user(user_id=e63897f83ea24e8e93ad52f4a68cb3bc)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:29.488730 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0200acc-b57c-4276-998f-27d1cc778b7b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:29.705693 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:29.707421 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d0200acc-b57c-4276-998f-27d1cc778b7b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e63897f83ea24e8e93ad52f4a68cb3bc'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:29.719310 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d0200acc-b57c-4276-998f-27d1cc778b7b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e63897f83ea24e8e93ad52f4a68cb3bc'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:29.830631 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d0200acc-b57c-4276-998f-27d1cc778b7b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e63897f83ea24e8e93ad52f4a68cb3bc'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:29.831353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d0200acc-b57c-4276-998f-27d1cc778b7b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e63897f83ea24e8e93ad52f4a68cb3bc'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:29.839767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d0200acc-b57c-4276-998f-27d1cc778b7b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e63897f83ea24e8e93ad52f4a68cb3bc'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:29.845645 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1274/2548] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:29 2022] DELETE /identity/v3/users/e63897f83ea24e8e93ad52f4a68cb3bc => generated 0 bytes in 431 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:30.305674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aff961c-ebd8-4a40-8856-2d435620131b admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:30.307225 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-8aff961c-ebd8-4a40-8856-2d435620131b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:30.323419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aff961c-ebd8-4a40-8856-2d435620131b admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:30.382442 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8980a90-30fb-457c-848c-2eff2c3301c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:30.383106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8980a90-30fb-457c-848c-2eff2c3301c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:30.383419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8980a90-30fb-457c-848c-2eff2c3301c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:30.383694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8980a90-30fb-457c-848c-2eff2c3301c3 admin admin] PATH_INFO: `/v3/projects/d564dfea143441c496cead0c412288c9` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:30.384800 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8980a90-30fb-457c-848c-2eff2c3301c3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d564dfea143441c496cead0c412288c9)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:30.387231 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8980a90-30fb-457c-848c-2eff2c3301c3 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:30.515013 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f8980a90-30fb-457c-848c-2eff2c3301c3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd564dfea143441c496cead0c412288c9', 'request_id': 'req-f8980a90-30fb-457c-848c-2eff2c3301c3'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:30.607737 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f8980a90-30fb-457c-848c-2eff2c3301c3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd564dfea143441c496cead0c412288c9', 'request_id': 'req-f8980a90-30fb-457c-848c-2eff2c3301c3'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:30.613905 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f8980a90-30fb-457c-848c-2eff2c3301c3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd564dfea143441c496cead0c412288c9', 'request_id': 'req-f8980a90-30fb-457c-848c-2eff2c3301c3'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:30.626146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f8980a90-30fb-457c-848c-2eff2c3301c3 admin admin] The token cache is being invalidate because project d564dfea143441c496cead0c412288c9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:30.626435 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f8980a90-30fb-457c-848c-2eff2c3301c3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:30.627121 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f8980a90-30fb-457c-848c-2eff2c3301c3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:30.628711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1275/2549] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:30 2022] DELETE /identity/v3/projects/d564dfea143441c496cead0c412288c9 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:30.633721 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c93096-2eba-4bdc-a01f-09fe36dfd1fe None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:30.636161 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c93096-2eba-4bdc-a01f-09fe36dfd1fe None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:30.636161 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c93096-2eba-4bdc-a01f-09fe36dfd1fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:30.659311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f1c93096-2eba-4bdc-a01f-09fe36dfd1fe None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:30.663671 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f1c93096-2eba-4bdc-a01f-09fe36dfd1fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:30.726951 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1275/2550] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:30.736492 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5699a370-3422-4700-9842-8ef016e05da1 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:30.737794 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5699a370-3422-4700-9842-8ef016e05da1 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:30.738107 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5699a370-3422-4700-9842-8ef016e05da1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:30.760539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-5699a370-3422-4700-9842-8ef016e05da1 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:30.761824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-5699a370-3422-4700-9842-8ef016e05da1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:30.770007 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1276/2551] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:56:30.774271 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c93096-2eba-4bdc-a01f-09fe36dfd1fe None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:30.785119 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c93096-2eba-4bdc-a01f-09fe36dfd1fe None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:30.790256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373e8287-bada-4d8e-bb8b-bfc77cdcb533 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:30.790639 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373e8287-bada-4d8e-bb8b-bfc77cdcb533 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:30.790848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373e8287-bada-4d8e-bb8b-bfc77cdcb533 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:30.791171 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373e8287-bada-4d8e-bb8b-bfc77cdcb533 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:30.791871 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373e8287-bada-4d8e-bb8b-bfc77cdcb533 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:56:30.792175 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373e8287-bada-4d8e-bb8b-bfc77cdcb533 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:30.794700 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373e8287-bada-4d8e-bb8b-bfc77cdcb533 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:30.809950 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1276/2552] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:56:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:56:30.814500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5699a370-3422-4700-9842-8ef016e05da1 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:30.823821 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5699a370-3422-4700-9842-8ef016e05da1 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:30.829175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b59559-7e0f-45f9-98c9-1d6e5c4715b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:30.829787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b59559-7e0f-45f9-98c9-1d6e5c4715b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:30.831207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b59559-7e0f-45f9-98c9-1d6e5c4715b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:30.831207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b59559-7e0f-45f9-98c9-1d6e5c4715b0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:30.831207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b59559-7e0f-45f9-98c9-1d6e5c4715b0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:30.833055 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b59559-7e0f-45f9-98c9-1d6e5c4715b0 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:30.910759 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1277/2553] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:30 2022] POST /identity/v3/projects => generated 381 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:30.911846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373e8287-bada-4d8e-bb8b-bfc77cdcb533 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:30.922800 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373e8287-bada-4d8e-bb8b-bfc77cdcb533 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:30.986895 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f753f295-83c8-47f6-9494-8cf6b2c49edd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:30.987567 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f753f295-83c8-47f6-9494-8cf6b2c49edd admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:30.987830 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f753f295-83c8-47f6-9494-8cf6b2c49edd admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:30.988103 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f753f295-83c8-47f6-9494-8cf6b2c49edd admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:30.988751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f753f295-83c8-47f6-9494-8cf6b2c49edd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:30.990829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f753f295-83c8-47f6-9494-8cf6b2c49edd admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:31.140631 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1277/2554] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:30 2022] POST /identity/v3/users => generated 291 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:31.145208 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b59559-7e0f-45f9-98c9-1d6e5c4715b0 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:31.156222 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b59559-7e0f-45f9-98c9-1d6e5c4715b0 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:31.161003 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ba7fe9-ab40-45bf-ae78-2635a986474a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:31.161549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ba7fe9-ab40-45bf-ae78-2635a986474a admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:31.161828 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ba7fe9-ab40-45bf-ae78-2635a986474a admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:31.162100 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ba7fe9-ab40-45bf-ae78-2635a986474a admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:31.162766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ba7fe9-ab40-45bf-ae78-2635a986474a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:31.164425 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ba7fe9-ab40-45bf-ae78-2635a986474a admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:31.175976 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1278/2555] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:31 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:31.180289 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f753f295-83c8-47f6-9494-8cf6b2c49edd admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:31.191756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f753f295-83c8-47f6-9494-8cf6b2c49edd admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:31.196438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dbb3d3-d82d-4a71-9172-afabc5106bfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:31.197039 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dbb3d3-d82d-4a71-9172-afabc5106bfe admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:31.197316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dbb3d3-d82d-4a71-9172-afabc5106bfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:31.197587 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dbb3d3-d82d-4a71-9172-afabc5106bfe admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:31.198281 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6dbb3d3-d82d-4a71-9172-afabc5106bfe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:31.199902 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6dbb3d3-d82d-4a71-9172-afabc5106bfe admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:31.211035 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1278/2556] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:31 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:31.215036 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ba7fe9-ab40-45bf-ae78-2635a986474a admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:31.225335 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ba7fe9-ab40-45bf-ae78-2635a986474a admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:31.230058 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25417d69-80a2-4feb-90a5-8dfed426718a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:31.230734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25417d69-80a2-4feb-90a5-8dfed426718a admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:31.231581 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25417d69-80a2-4feb-90a5-8dfed426718a admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:31.231871 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25417d69-80a2-4feb-90a5-8dfed426718a admin admin] PATH_INFO: `/v3/projects/816c1a663e924bce8ed77005cb9b20a9/users/193cb8ffbade4f3489d17abd2dfe9761/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:31.243103 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:DELETE=12,keystone:SELECT=85,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:31.247593 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25417d69-80a2-4feb-90a5-8dfed426718a admin admin] RBAC: Authorizing `identity:create_grant(project_id=816c1a663e924bce8ed77005cb9b20a9, user_id=193cb8ffbade4f3489d17abd2dfe9761, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:31.250251 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25417d69-80a2-4feb-90a5-8dfed426718a admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:31.408283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1279/2557] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:31 2022] PUT /identity/v3/projects/816c1a663e924bce8ed77005cb9b20a9/users/193cb8ffbade4f3489d17abd2dfe9761/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:31.418847 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e1b124-06e8-4a21-8b2d-ba19c5214b25 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:31.419104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e1b124-06e8-4a21-8b2d-ba19c5214b25 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:31.419378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e1b124-06e8-4a21-8b2d-ba19c5214b25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:31.447357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-02e1b124-06e8-4a21-8b2d-ba19c5214b25 None None] MFA Rules not processed for user `193cb8ffbade4f3489d17abd2dfe9761`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:31.448554 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-02e1b124-06e8-4a21-8b2d-ba19c5214b25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:31.466852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=93,keystone:DELETE=3,keystone:INSERT=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:31.565885 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25417d69-80a2-4feb-90a5-8dfed426718a admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:31.566985 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1279/2558] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:31 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:31.690799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25417d69-80a2-4feb-90a5-8dfed426718a admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:31.696591 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299e6eb5-756f-4013-a9a3-558698a049fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:31.697647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299e6eb5-756f-4013-a9a3-558698a049fb admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:31.697647 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299e6eb5-756f-4013-a9a3-558698a049fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:31.698276 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299e6eb5-756f-4013-a9a3-558698a049fb admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:31.698992 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299e6eb5-756f-4013-a9a3-558698a049fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:31.701403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299e6eb5-756f-4013-a9a3-558698a049fb admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:31.829781 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1280/2559] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:31 2022] POST /identity/v3/projects => generated 387 bytes in 264 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:31.835433 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e1b124-06e8-4a21-8b2d-ba19c5214b25 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:31.865126 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e1b124-06e8-4a21-8b2d-ba19c5214b25 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:31.940061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bc7153-3c62-4210-8aea-5d6548867f42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:31.940594 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bc7153-3c62-4210-8aea-5d6548867f42 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:31.940870 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bc7153-3c62-4210-8aea-5d6548867f42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:31.941140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bc7153-3c62-4210-8aea-5d6548867f42 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:31.941769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4bc7153-3c62-4210-8aea-5d6548867f42 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:31.943748 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4bc7153-3c62-4210-8aea-5d6548867f42 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:32.038293 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1280/2560] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:31 2022] POST /identity/v3/users => generated 300 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:32.042438 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299e6eb5-756f-4013-a9a3-558698a049fb admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:32.053802 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299e6eb5-756f-4013-a9a3-558698a049fb admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:32.062203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f40d10d-0e7d-43d6-9f54-101cbcbb9a7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:32.062827 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f40d10d-0e7d-43d6-9f54-101cbcbb9a7c admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:32.063173 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f40d10d-0e7d-43d6-9f54-101cbcbb9a7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:32.063446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f40d10d-0e7d-43d6-9f54-101cbcbb9a7c admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:32.064089 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f40d10d-0e7d-43d6-9f54-101cbcbb9a7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:32.065705 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f40d10d-0e7d-43d6-9f54-101cbcbb9a7c admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:32.078240 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1281/2561] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:32 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:32.082731 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bc7153-3c62-4210-8aea-5d6548867f42 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:32.093229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bc7153-3c62-4210-8aea-5d6548867f42 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:32.098956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de169385-2897-4267-95ed-b60f6def0dde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:32.099445 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de169385-2897-4267-95ed-b60f6def0dde admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:32.099725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de169385-2897-4267-95ed-b60f6def0dde admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:32.099998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de169385-2897-4267-95ed-b60f6def0dde admin admin] PATH_INFO: `/v3/projects/2d777a6283fd49d0868014430654788b/users/4f00cb4c2737450b92e061d9a75e6c03/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:32.120790 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de169385-2897-4267-95ed-b60f6def0dde admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d777a6283fd49d0868014430654788b, user_id=4f00cb4c2737450b92e061d9a75e6c03, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:32.124282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de169385-2897-4267-95ed-b60f6def0dde admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:32.225363 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1281/2562] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:32 2022] PUT /identity/v3/projects/2d777a6283fd49d0868014430654788b/users/4f00cb4c2737450b92e061d9a75e6c03/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:56:32.229347 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f40d10d-0e7d-43d6-9f54-101cbcbb9a7c admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:32.239403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f40d10d-0e7d-43d6-9f54-101cbcbb9a7c admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:32.299468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18de292-8d18-4df8-a3d8-ff41808bfa79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:32.300437 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18de292-8d18-4df8-a3d8-ff41808bfa79 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:32.300739 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18de292-8d18-4df8-a3d8-ff41808bfa79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:32.301013 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18de292-8d18-4df8-a3d8-ff41808bfa79 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:32.301676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18de292-8d18-4df8-a3d8-ff41808bfa79 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:32.303317 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18de292-8d18-4df8-a3d8-ff41808bfa79 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:32.318960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1282/2563] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:32 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:32.320904 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de169385-2897-4267-95ed-b60f6def0dde admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:32.334131 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de169385-2897-4267-95ed-b60f6def0dde admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:32.343364 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ad46f6-e4ce-4770-b8dc-202685c74550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:32.344482 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ad46f6-e4ce-4770-b8dc-202685c74550 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:32.344482 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ad46f6-e4ce-4770-b8dc-202685c74550 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:32.348722 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ad46f6-e4ce-4770-b8dc-202685c74550 admin admin] PATH_INFO: `/v3/projects/2d777a6283fd49d0868014430654788b/users/4f00cb4c2737450b92e061d9a75e6c03/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:32.350884 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ad46f6-e4ce-4770-b8dc-202685c74550 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d777a6283fd49d0868014430654788b, user_id=4f00cb4c2737450b92e061d9a75e6c03, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:32.353563 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ad46f6-e4ce-4770-b8dc-202685c74550 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:32.366764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1282/2564] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:32 2022] PUT /identity/v3/projects/2d777a6283fd49d0868014430654788b/users/4f00cb4c2737450b92e061d9a75e6c03/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:32.383713 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca2a57d-775c-4d12-8aa6-00de4671d360 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:32.384100 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca2a57d-775c-4d12-8aa6-00de4671d360 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:32.384821 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca2a57d-775c-4d12-8aa6-00de4671d360 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:32.420489 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-1ca2a57d-775c-4d12-8aa6-00de4671d360 None None] MFA Rules not processed for user `4f00cb4c2737450b92e061d9a75e6c03`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:32.421704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1ca2a57d-775c-4d12-8aa6-00de4671d360 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:32.489182 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:32.510770 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1283/2565] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:32 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:32.523126 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ad46f6-e4ce-4770-b8dc-202685c74550 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:32.523399 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-35ad46f6-e4ce-4770-b8dc-202685c74550 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:32.583826 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ad46f6-e4ce-4770-b8dc-202685c74550 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:32.588008 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb3a234-196e-4e83-8391-ee08d5e966c3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:32.592182 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb3a234-196e-4e83-8391-ee08d5e966c3 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:32.592182 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb3a234-196e-4e83-8391-ee08d5e966c3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:32.592182 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb3a234-196e-4e83-8391-ee08d5e966c3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:32.602719 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb3a234-196e-4e83-8391-ee08d5e966c3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:32.604288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb3a234-196e-4e83-8391-ee08d5e966c3 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:32.623030 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1283/2566] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:32 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:32.714403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca2a57d-775c-4d12-8aa6-00de4671d360 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:32.716409 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1ca2a57d-775c-4d12-8aa6-00de4671d360 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:32.789188 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca2a57d-775c-4d12-8aa6-00de4671d360 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:32.794310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa27d0d-ad46-42a0-8737-52c872c42952 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:32.795045 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa27d0d-ad46-42a0-8737-52c872c42952 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:32.795327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa27d0d-ad46-42a0-8737-52c872c42952 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:32.795599 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa27d0d-ad46-42a0-8737-52c872c42952 admin admin] PATH_INFO: `/v3/users/56c6088d1a0c4e68ad0e2235a8d5981d` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:32.796728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa27d0d-ad46-42a0-8737-52c872c42952 admin admin] RBAC: Authorizing `identity:delete_user(user_id=56c6088d1a0c4e68ad0e2235a8d5981d)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:32.798646 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa27d0d-ad46-42a0-8737-52c872c42952 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:33.032152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1aa27d0d-ad46-42a0-8737-52c872c42952 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '56c6088d1a0c4e68ad0e2235a8d5981d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:33.110851 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1aa27d0d-ad46-42a0-8737-52c872c42952 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '56c6088d1a0c4e68ad0e2235a8d5981d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:33.146420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1aa27d0d-ad46-42a0-8737-52c872c42952 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '56c6088d1a0c4e68ad0e2235a8d5981d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:33.146420 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1aa27d0d-ad46-42a0-8737-52c872c42952 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '56c6088d1a0c4e68ad0e2235a8d5981d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:33.163805 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1aa27d0d-ad46-42a0-8737-52c872c42952 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '56c6088d1a0c4e68ad0e2235a8d5981d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:33.174344 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1284/2567] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:32 2022] DELETE /identity/v3/users/56c6088d1a0c4e68ad0e2235a8d5981d => generated 0 bytes in 460 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:33.178733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c852745-2534-4ddc-a3ae-85b56a552d3c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:33.184803 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c852745-2534-4ddc-a3ae-85b56a552d3c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:33.184803 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c852745-2534-4ddc-a3ae-85b56a552d3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:33.204542 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-6c852745-2534-4ddc-a3ae-85b56a552d3c None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:33.206151 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6c852745-2534-4ddc-a3ae-85b56a552d3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:33.220165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:33.292937 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1284/2568] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:33.299352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa27d0d-ad46-42a0-8737-52c872c42952 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:33.301355 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1aa27d0d-ad46-42a0-8737-52c872c42952 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:33.371310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa27d0d-ad46-42a0-8737-52c872c42952 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:33.375024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf27411-64f2-467b-b9de-caa32e323ed3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:33.375665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf27411-64f2-467b-b9de-caa32e323ed3 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:33.375949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf27411-64f2-467b-b9de-caa32e323ed3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:33.376226 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf27411-64f2-467b-b9de-caa32e323ed3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:33.386358 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf27411-64f2-467b-b9de-caa32e323ed3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:33.389021 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf27411-64f2-467b-b9de-caa32e323ed3 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:33.408860 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1285/2569] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:33.994984 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c852745-2534-4ddc-a3ae-85b56a552d3c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:33.996488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6c852745-2534-4ddc-a3ae-85b56a552d3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:34.009229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c852745-2534-4ddc-a3ae-85b56a552d3c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:34.017596 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf27411-64f2-467b-b9de-caa32e323ed3 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:34.018103 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0364e6-0f4f-418d-8d38-fa52827a315c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:34.018103 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0364e6-0f4f-418d-8d38-fa52827a315c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:34.018103 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0364e6-0f4f-418d-8d38-fa52827a315c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:34.018103 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0364e6-0f4f-418d-8d38-fa52827a315c admin admin] PATH_INFO: `/v3/users/193cb8ffbade4f3489d17abd2dfe9761` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:34.019295 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-dbf27411-64f2-467b-b9de-caa32e323ed3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:34.019441 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0364e6-0f4f-418d-8d38-fa52827a315c admin admin] RBAC: Authorizing `identity:delete_user(user_id=193cb8ffbade4f3489d17abd2dfe9761)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:34.021876 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0364e6-0f4f-418d-8d38-fa52827a315c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:34.037698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf27411-64f2-467b-b9de-caa32e323ed3 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:34.046876 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9c043b-4770-427a-a131-f82d4a0cdfba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:34.046876 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9c043b-4770-427a-a131-f82d4a0cdfba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:34.047606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9c043b-4770-427a-a131-f82d4a0cdfba admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:34.047904 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9c043b-4770-427a-a131-f82d4a0cdfba admin admin] PATH_INFO: `/v3/projects/f4dbda910bbd48d997a9896c77880f87` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:34.049097 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9c043b-4770-427a-a131-f82d4a0cdfba admin admin] RBAC: Authorizing `identity:delete_project(project_id=f4dbda910bbd48d997a9896c77880f87)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:34.051576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9c043b-4770-427a-a131-f82d4a0cdfba admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:34.053537 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:34.255134 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-4e9c043b-4770-427a-a131-f82d4a0cdfba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f4dbda910bbd48d997a9896c77880f87', 'request_id': 'req-4e9c043b-4770-427a-a131-f82d4a0cdfba'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:34.271095 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-cf0364e6-0f4f-418d-8d38-fa52827a315c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '193cb8ffbade4f3489d17abd2dfe9761'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:34.285146 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-cf0364e6-0f4f-418d-8d38-fa52827a315c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '193cb8ffbade4f3489d17abd2dfe9761'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:34.460767 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-4e9c043b-4770-427a-a131-f82d4a0cdfba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f4dbda910bbd48d997a9896c77880f87', 'request_id': 'req-4e9c043b-4770-427a-a131-f82d4a0cdfba'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:34.470221 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-4e9c043b-4770-427a-a131-f82d4a0cdfba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f4dbda910bbd48d997a9896c77880f87', 'request_id': 'req-4e9c043b-4770-427a-a131-f82d4a0cdfba'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:34.484149 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-4e9c043b-4770-427a-a131-f82d4a0cdfba admin admin] The token cache is being invalidate because project f4dbda910bbd48d997a9896c77880f87 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:34.484673 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-4e9c043b-4770-427a-a131-f82d4a0cdfba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:34.485698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-4e9c043b-4770-427a-a131-f82d4a0cdfba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:34.487472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1286/2570] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:34 2022] DELETE /identity/v3/projects/f4dbda910bbd48d997a9896c77880f87 => generated 0 bytes in 472 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:34.491553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9c043b-4770-427a-a131-f82d4a0cdfba admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:34.493099 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-4e9c043b-4770-427a-a131-f82d4a0cdfba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:34.514077 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-cf0364e6-0f4f-418d-8d38-fa52827a315c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '193cb8ffbade4f3489d17abd2dfe9761'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:34.514942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-cf0364e6-0f4f-418d-8d38-fa52827a315c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '193cb8ffbade4f3489d17abd2dfe9761'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:34.530395 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-cf0364e6-0f4f-418d-8d38-fa52827a315c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '193cb8ffbade4f3489d17abd2dfe9761'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:34.551918 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1285/2571] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:33 2022] DELETE /identity/v3/users/193cb8ffbade4f3489d17abd2dfe9761 => generated 0 bytes in 558 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:34.564421 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab42fb88-9096-4723-847f-4519fe5d46e4 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:34.564780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab42fb88-9096-4723-847f-4519fe5d46e4 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:34.565159 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab42fb88-9096-4723-847f-4519fe5d46e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:34.566864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9c043b-4770-427a-a131-f82d4a0cdfba admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:34.588237 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ab42fb88-9096-4723-847f-4519fe5d46e4 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:34.589610 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ab42fb88-9096-4723-847f-4519fe5d46e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:34.639701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d98569-62bf-447e-abff-b8294c9c90c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:34.641899 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d98569-62bf-447e-abff-b8294c9c90c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:34.642362 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d98569-62bf-447e-abff-b8294c9c90c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:34.642798 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d98569-62bf-447e-abff-b8294c9c90c2 admin admin] PATH_INFO: `/v3/users/fb7cc9ae0d4241abba10f65093fa7f06` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:34.644194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d98569-62bf-447e-abff-b8294c9c90c2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fb7cc9ae0d4241abba10f65093fa7f06)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:34.646414 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d98569-62bf-447e-abff-b8294c9c90c2 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:34.822926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1286/2572] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:56:34.823692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab42fb88-9096-4723-847f-4519fe5d46e4 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:34.824982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ab42fb88-9096-4723-847f-4519fe5d46e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:34.836067 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab42fb88-9096-4723-847f-4519fe5d46e4 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:34.841102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c44966-339f-4d92-9b00-352e63830e32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:34.841710 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c44966-339f-4d92-9b00-352e63830e32 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:34.841985 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c44966-339f-4d92-9b00-352e63830e32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:34.842256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c44966-339f-4d92-9b00-352e63830e32 admin admin] PATH_INFO: `/v3/users/cbae7d1df881469f97348d037e3d2ec2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:34.843404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c44966-339f-4d92-9b00-352e63830e32 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cbae7d1df881469f97348d037e3d2ec2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:34.845275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c44966-339f-4d92-9b00-352e63830e32 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:34.867404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:34.890922 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:34.896823 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-31d98569-62bf-447e-abff-b8294c9c90c2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fb7cc9ae0d4241abba10f65093fa7f06'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:34.985050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-31d98569-62bf-447e-abff-b8294c9c90c2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fb7cc9ae0d4241abba10f65093fa7f06'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:34.998037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-31d98569-62bf-447e-abff-b8294c9c90c2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fb7cc9ae0d4241abba10f65093fa7f06'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:34.998726 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-31d98569-62bf-447e-abff-b8294c9c90c2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fb7cc9ae0d4241abba10f65093fa7f06'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:35.005681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-31d98569-62bf-447e-abff-b8294c9c90c2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fb7cc9ae0d4241abba10f65093fa7f06'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:35.011485 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1287/2573] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:34 2022] DELETE /identity/v3/users/fb7cc9ae0d4241abba10f65093fa7f06 => generated 0 bytes in 520 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:56:35.015153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d98569-62bf-447e-abff-b8294c9c90c2 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:35.015153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-31d98569-62bf-447e-abff-b8294c9c90c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:35.044962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-63c44966-339f-4d92-9b00-352e63830e32 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cbae7d1df881469f97348d037e3d2ec2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:35.062466 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-63c44966-339f-4d92-9b00-352e63830e32 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cbae7d1df881469f97348d037e3d2ec2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:35.080137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d98569-62bf-447e-abff-b8294c9c90c2 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:35.083939 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f99ea6-8021-41b2-8f7b-27e6c3a875d0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:35.084550 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f99ea6-8021-41b2-8f7b-27e6c3a875d0 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:35.084832 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f99ea6-8021-41b2-8f7b-27e6c3a875d0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:35.085105 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f99ea6-8021-41b2-8f7b-27e6c3a875d0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:35.087025 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-87f99ea6-8021-41b2-8f7b-27e6c3a875d0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:35.139456 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-63c44966-339f-4d92-9b00-352e63830e32 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cbae7d1df881469f97348d037e3d2ec2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:35.140107 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-63c44966-339f-4d92-9b00-352e63830e32 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cbae7d1df881469f97348d037e3d2ec2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:35.147253 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-63c44966-339f-4d92-9b00-352e63830e32 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cbae7d1df881469f97348d037e3d2ec2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:35.165218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1287/2574] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:34 2022] DELETE /identity/v3/users/cbae7d1df881469f97348d037e3d2ec2 => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:35.165395 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f99ea6-8021-41b2-8f7b-27e6c3a875d0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:35.165395 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f99ea6-8021-41b2-8f7b-27e6c3a875d0 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:35.165607 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641d18ae-349f-48bb-ad63-bc6d0dc51bfe None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:35.165607 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641d18ae-349f-48bb-ad63-bc6d0dc51bfe None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:35.165607 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641d18ae-349f-48bb-ad63-bc6d0dc51bfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:35.166000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1288/2575] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:35.178157 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-641d18ae-349f-48bb-ad63-bc6d0dc51bfe None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:35.179518 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-641d18ae-349f-48bb-ad63-bc6d0dc51bfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:35.241310 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1288/2576] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:56:35.251511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f99ea6-8021-41b2-8f7b-27e6c3a875d0 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:35.260126 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f99ea6-8021-41b2-8f7b-27e6c3a875d0 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:35.320453 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1e3cbd-8b10-4233-b092-e755d3a0c9b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:35.321327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1e3cbd-8b10-4233-b092-e755d3a0c9b0 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:35.322058 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1e3cbd-8b10-4233-b092-e755d3a0c9b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:35.322058 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1e3cbd-8b10-4233-b092-e755d3a0c9b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:35.333464 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1e3cbd-8b10-4233-b092-e755d3a0c9b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:35.335110 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1e3cbd-8b10-4233-b092-e755d3a0c9b0 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:35.347135 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1289/2577] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:35.644547 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641d18ae-349f-48bb-ad63-bc6d0dc51bfe None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:35.646653 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-641d18ae-349f-48bb-ad63-bc6d0dc51bfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:35.659539 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641d18ae-349f-48bb-ad63-bc6d0dc51bfe None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:35.682500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a248e51-d630-4623-9754-287f62c2ae1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:35.683590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a248e51-d630-4623-9754-287f62c2ae1c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:35.684868 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a248e51-d630-4623-9754-287f62c2ae1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:35.685705 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a248e51-d630-4623-9754-287f62c2ae1c admin admin] PATH_INFO: `/v3/projects/df4eb1d84244463397cd5c153d7ecc74` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:35.687141 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a248e51-d630-4623-9754-287f62c2ae1c admin admin] RBAC: Authorizing `identity:delete_project(project_id=df4eb1d84244463397cd5c153d7ecc74)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:35.690630 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a248e51-d630-4623-9754-287f62c2ae1c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:35.811251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6a248e51-d630-4623-9754-287f62c2ae1c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df4eb1d84244463397cd5c153d7ecc74', 'request_id': 'req-6a248e51-d630-4623-9754-287f62c2ae1c'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:35.819098 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6a248e51-d630-4623-9754-287f62c2ae1c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df4eb1d84244463397cd5c153d7ecc74', 'request_id': 'req-6a248e51-d630-4623-9754-287f62c2ae1c'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:35.904976 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6a248e51-d630-4623-9754-287f62c2ae1c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df4eb1d84244463397cd5c153d7ecc74', 'request_id': 'req-6a248e51-d630-4623-9754-287f62c2ae1c'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:35.918600 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6a248e51-d630-4623-9754-287f62c2ae1c admin admin] The token cache is being invalidate because project df4eb1d84244463397cd5c153d7ecc74 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:35.918600 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6a248e51-d630-4623-9754-287f62c2ae1c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:35.918600 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6a248e51-d630-4623-9754-287f62c2ae1c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:35.919165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1289/2578] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:35 2022] DELETE /identity/v3/projects/df4eb1d84244463397cd5c153d7ecc74 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:35.923816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd439b9-76d8-4197-aec2-e4dd92af7da8 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:35.924103 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd439b9-76d8-4197-aec2-e4dd92af7da8 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:35.924374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd439b9-76d8-4197-aec2-e4dd92af7da8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:35.948158 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-8fd439b9-76d8-4197-aec2-e4dd92af7da8 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:35.949346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-8fd439b9-76d8-4197-aec2-e4dd92af7da8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:35.958616 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:DELETE=5,keystone:SELECT=94,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:35.988544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a248e51-d630-4623-9754-287f62c2ae1c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:35.990046 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6a248e51-d630-4623-9754-287f62c2ae1c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:36.017306 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1290/2579] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:36.019848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:36.025816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59557530-7289-493d-aa9f-46b1137c1629 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:36.026215 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59557530-7289-493d-aa9f-46b1137c1629 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:36.026588 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59557530-7289-493d-aa9f-46b1137c1629 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:36.047277 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-59557530-7289-493d-aa9f-46b1137c1629 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:36.048228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-59557530-7289-493d-aa9f-46b1137c1629 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:36.057111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1291/2580] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:56:36.058258 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59557530-7289-493d-aa9f-46b1137c1629 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:36.060785 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-59557530-7289-493d-aa9f-46b1137c1629 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:36.069278 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a248e51-d630-4623-9754-287f62c2ae1c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:36.075618 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59557530-7289-493d-aa9f-46b1137c1629 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:36.077942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f18a5f-6bb2-4734-a76b-4e4f04ca5210 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:36.078947 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f18a5f-6bb2-4734-a76b-4e4f04ca5210 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:36.085385 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f18a5f-6bb2-4734-a76b-4e4f04ca5210 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:36.085385 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f18a5f-6bb2-4734-a76b-4e4f04ca5210 admin admin] PATH_INFO: `/v3/projects/816c1a663e924bce8ed77005cb9b20a9` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:36.085385 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f18a5f-6bb2-4734-a76b-4e4f04ca5210 admin admin] RBAC: Authorizing `identity:delete_project(project_id=816c1a663e924bce8ed77005cb9b20a9)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:36.085385 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f18a5f-6bb2-4734-a76b-4e4f04ca5210 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:36.086318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c99fd8-b597-4878-97d7-99028b97c6ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:36.086318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c99fd8-b597-4878-97d7-99028b97c6ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:36.086318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c99fd8-b597-4878-97d7-99028b97c6ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:36.086318 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c99fd8-b597-4878-97d7-99028b97c6ea admin admin] PATH_INFO: `/v3/projects/fb7f1e80fe074809bc81d3b9c98d0e2e` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:36.087498 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c99fd8-b597-4878-97d7-99028b97c6ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb7f1e80fe074809bc81d3b9c98d0e2e)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:36.090391 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c99fd8-b597-4878-97d7-99028b97c6ea admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:36.227149 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-16c99fd8-b597-4878-97d7-99028b97c6ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb7f1e80fe074809bc81d3b9c98d0e2e', 'request_id': 'req-16c99fd8-b597-4878-97d7-99028b97c6ea'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:36.229698 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-94f18a5f-6bb2-4734-a76b-4e4f04ca5210 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '816c1a663e924bce8ed77005cb9b20a9', 'request_id': 'req-94f18a5f-6bb2-4734-a76b-4e4f04ca5210'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:36.236682 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-94f18a5f-6bb2-4734-a76b-4e4f04ca5210 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '816c1a663e924bce8ed77005cb9b20a9', 'request_id': 'req-94f18a5f-6bb2-4734-a76b-4e4f04ca5210'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:36.361810 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-94f18a5f-6bb2-4734-a76b-4e4f04ca5210 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '816c1a663e924bce8ed77005cb9b20a9', 'request_id': 'req-94f18a5f-6bb2-4734-a76b-4e4f04ca5210'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:36.373001 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-94f18a5f-6bb2-4734-a76b-4e4f04ca5210 admin admin] The token cache is being invalidate because project 816c1a663e924bce8ed77005cb9b20a9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:36.373297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-94f18a5f-6bb2-4734-a76b-4e4f04ca5210 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:36.373971 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-94f18a5f-6bb2-4734-a76b-4e4f04ca5210 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:36.375512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1290/2581] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:35 2022] DELETE /identity/v3/projects/816c1a663e924bce8ed77005cb9b20a9 => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:36.376264 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f18a5f-6bb2-4734-a76b-4e4f04ca5210 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:36.377595 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-94f18a5f-6bb2-4734-a76b-4e4f04ca5210 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:36.419416 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-16c99fd8-b597-4878-97d7-99028b97c6ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb7f1e80fe074809bc81d3b9c98d0e2e', 'request_id': 'req-16c99fd8-b597-4878-97d7-99028b97c6ea'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:36.425836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-16c99fd8-b597-4878-97d7-99028b97c6ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb7f1e80fe074809bc81d3b9c98d0e2e', 'request_id': 'req-16c99fd8-b597-4878-97d7-99028b97c6ea'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:36.438441 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-16c99fd8-b597-4878-97d7-99028b97c6ea admin admin] The token cache is being invalidate because project fb7f1e80fe074809bc81d3b9c98d0e2e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:36.438665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-16c99fd8-b597-4878-97d7-99028b97c6ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:36.439340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-16c99fd8-b597-4878-97d7-99028b97c6ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:36.441015 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1292/2582] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:36 2022] DELETE /identity/v3/projects/fb7f1e80fe074809bc81d3b9c98d0e2e => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:36.441888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c99fd8-b597-4878-97d7-99028b97c6ea admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:36.443295 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-16c99fd8-b597-4878-97d7-99028b97c6ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:36.448733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f18a5f-6bb2-4734-a76b-4e4f04ca5210 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:36.454515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851a612e-9b5d-4d53-81c0-6935148d45a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:36.455218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851a612e-9b5d-4d53-81c0-6935148d45a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:36.455498 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851a612e-9b5d-4d53-81c0-6935148d45a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:36.455770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851a612e-9b5d-4d53-81c0-6935148d45a0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:36.456414 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851a612e-9b5d-4d53-81c0-6935148d45a0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:56:36.456878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851a612e-9b5d-4d53-81c0-6935148d45a0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:36.458404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851a612e-9b5d-4d53-81c0-6935148d45a0 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:36.478410 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1291/2583] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:56:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:56:36.478410 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf9b08f-7b0b-4c2a-8c5f-f3583cf70e96 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:36.478410 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf9b08f-7b0b-4c2a-8c5f-f3583cf70e96 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:36.478410 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf9b08f-7b0b-4c2a-8c5f-f3583cf70e96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:36.497626 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2cf9b08f-7b0b-4c2a-8c5f-f3583cf70e96 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:36.498847 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2cf9b08f-7b0b-4c2a-8c5f-f3583cf70e96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:36.506926 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1292/2584] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:36.507761 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf9b08f-7b0b-4c2a-8c5f-f3583cf70e96 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:36.509006 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2cf9b08f-7b0b-4c2a-8c5f-f3583cf70e96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:36.513743 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c99fd8-b597-4878-97d7-99028b97c6ea admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:36.518659 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f040953-93fc-4ca8-b9a6-15b73728238f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:36.519428 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f040953-93fc-4ca8-b9a6-15b73728238f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:36.519787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f040953-93fc-4ca8-b9a6-15b73728238f admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:36.520067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f040953-93fc-4ca8-b9a6-15b73728238f admin admin] PATH_INFO: `/v3/users/4f00cb4c2737450b92e061d9a75e6c03` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:36.521212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f040953-93fc-4ca8-b9a6-15b73728238f admin admin] RBAC: Authorizing `identity:delete_user(user_id=4f00cb4c2737450b92e061d9a75e6c03)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:36.521588 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf9b08f-7b0b-4c2a-8c5f-f3583cf70e96 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:36.523183 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f040953-93fc-4ca8-b9a6-15b73728238f admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:36.526288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a92e10e-5bac-4afe-b182-bd92953e041b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:36.526880 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a92e10e-5bac-4afe-b182-bd92953e041b admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:36.527194 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a92e10e-5bac-4afe-b182-bd92953e041b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:36.527457 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a92e10e-5bac-4afe-b182-bd92953e041b admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:36.528681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a92e10e-5bac-4afe-b182-bd92953e041b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:36.530578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a92e10e-5bac-4afe-b182-bd92953e041b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:36.648721 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1293/2585] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:36 2022] POST /identity/v3/projects => generated 375 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:36.650024 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844f4875-04d0-4038-a107-43a5b9031aae None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:36.650312 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844f4875-04d0-4038-a107-43a5b9031aae None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:36.650590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844f4875-04d0-4038-a107-43a5b9031aae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:36.672195 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-844f4875-04d0-4038-a107-43a5b9031aae None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:36.673409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-844f4875-04d0-4038-a107-43a5b9031aae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:36.731847 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1294/2586] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:36.732660 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844f4875-04d0-4038-a107-43a5b9031aae None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:36.741757 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844f4875-04d0-4038-a107-43a5b9031aae None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:36.746391 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44721a6-8e45-49d7-89fc-9a4c012f90d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:36.747553 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44721a6-8e45-49d7-89fc-9a4c012f90d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:36.747822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44721a6-8e45-49d7-89fc-9a4c012f90d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:36.748087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44721a6-8e45-49d7-89fc-9a4c012f90d5 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:36.748702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44721a6-8e45-49d7-89fc-9a4c012f90d5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:36.750017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1f040953-93fc-4ca8-b9a6-15b73728238f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4f00cb4c2737450b92e061d9a75e6c03'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:36.750581 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44721a6-8e45-49d7-89fc-9a4c012f90d5 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:36.836147 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1f040953-93fc-4ca8-b9a6-15b73728238f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4f00cb4c2737450b92e061d9a75e6c03'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:36.848782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1f040953-93fc-4ca8-b9a6-15b73728238f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4f00cb4c2737450b92e061d9a75e6c03'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:36.849416 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1f040953-93fc-4ca8-b9a6-15b73728238f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4f00cb4c2737450b92e061d9a75e6c03'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:36.856633 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1f040953-93fc-4ca8-b9a6-15b73728238f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4f00cb4c2737450b92e061d9a75e6c03'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:36.862462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1293/2587] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:36 2022] DELETE /identity/v3/users/4f00cb4c2737450b92e061d9a75e6c03 => generated 0 bytes in 421 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:36.863385 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f040953-93fc-4ca8-b9a6-15b73728238f admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:36.864704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1f040953-93fc-4ca8-b9a6-15b73728238f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:36.875837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1295/2588] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:36 2022] POST /identity/v3/users => generated 288 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:36.882185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44721a6-8e45-49d7-89fc-9a4c012f90d5 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:36.884481 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d44721a6-8e45-49d7-89fc-9a4c012f90d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:36.898712 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44721a6-8e45-49d7-89fc-9a4c012f90d5 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:36.933070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f040953-93fc-4ca8-b9a6-15b73728238f admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:36.939159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ed6f58-5e5e-4392-b1a5-91ba8f939373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:36.939779 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ed6f58-5e5e-4392-b1a5-91ba8f939373 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:36.940063 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ed6f58-5e5e-4392-b1a5-91ba8f939373 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:36.940340 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ed6f58-5e5e-4392-b1a5-91ba8f939373 admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:36.940988 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ed6f58-5e5e-4392-b1a5-91ba8f939373 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:56:36.941287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ed6f58-5e5e-4392-b1a5-91ba8f939373 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:36.943109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ed6f58-5e5e-4392-b1a5-91ba8f939373 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:36.955828 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1294/2589] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:56:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:56:36.960279 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ed6f58-5e5e-4392-b1a5-91ba8f939373 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:36.971269 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ed6f58-5e5e-4392-b1a5-91ba8f939373 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:36.973122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9562902-865a-49a0-b2c0-6be3b4f0d781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:36.973740 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9562902-865a-49a0-b2c0-6be3b4f0d781 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:36.974064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9562902-865a-49a0-b2c0-6be3b4f0d781 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:36.974446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9562902-865a-49a0-b2c0-6be3b4f0d781 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:36.975124 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9562902-865a-49a0-b2c0-6be3b4f0d781 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:36.975866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6bff78-efba-4d51-b068-eee38e297472 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:36.976460 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6bff78-efba-4d51-b068-eee38e297472 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:36.976725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9562902-865a-49a0-b2c0-6be3b4f0d781 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:36.976856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6bff78-efba-4d51-b068-eee38e297472 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:36.977000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6bff78-efba-4d51-b068-eee38e297472 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:36.977658 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6bff78-efba-4d51-b068-eee38e297472 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:36.979568 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6bff78-efba-4d51-b068-eee38e297472 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:36.987652 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1296/2590] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:36 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:36.991631 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9562902-865a-49a0-b2c0-6be3b4f0d781 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:37.001898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9562902-865a-49a0-b2c0-6be3b4f0d781 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:37.006698 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9816ce9-ea65-4b46-b1b2-335291b41b99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:37.007310 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9816ce9-ea65-4b46-b1b2-335291b41b99 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:37.007585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9816ce9-ea65-4b46-b1b2-335291b41b99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:37.007854 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9816ce9-ea65-4b46-b1b2-335291b41b99 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:37.008472 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9816ce9-ea65-4b46-b1b2-335291b41b99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:37.010050 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9816ce9-ea65-4b46-b1b2-335291b41b99 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:37.020942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1297/2591] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:56:37.024863 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9816ce9-ea65-4b46-b1b2-335291b41b99 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:37.034309 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9816ce9-ea65-4b46-b1b2-335291b41b99 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:37.038803 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c90a4e-5ed9-4a5a-887e-56e1fcbab95a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:37.042989 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c90a4e-5ed9-4a5a-887e-56e1fcbab95a admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:37.042989 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c90a4e-5ed9-4a5a-887e-56e1fcbab95a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:37.042989 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c90a4e-5ed9-4a5a-887e-56e1fcbab95a admin admin] PATH_INFO: `/v3/projects/a9aebb23f7c644b1b36f4d19154490e0/users/b970f9cb8fe444f3a25fac913f23f84d/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:37.054044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c90a4e-5ed9-4a5a-887e-56e1fcbab95a admin admin] RBAC: Authorizing `identity:create_grant(project_id=a9aebb23f7c644b1b36f4d19154490e0, user_id=b970f9cb8fe444f3a25fac913f23f84d, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:37.056685 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c90a4e-5ed9-4a5a-887e-56e1fcbab95a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:37.098526 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1295/2592] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:36 2022] POST /identity/v3/projects => generated 403 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:56:37.099607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6bff78-efba-4d51-b068-eee38e297472 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:37.169906 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6bff78-efba-4d51-b068-eee38e297472 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:37.174796 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251667aa-7406-4043-80aa-f785580c8425 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:37.175436 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251667aa-7406-4043-80aa-f785580c8425 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:37.175715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251667aa-7406-4043-80aa-f785580c8425 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:37.175982 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251667aa-7406-4043-80aa-f785580c8425 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:37.176604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251667aa-7406-4043-80aa-f785580c8425 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:37.178768 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251667aa-7406-4043-80aa-f785580c8425 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:37.192315 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1298/2593] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:37 2022] PUT /identity/v3/projects/a9aebb23f7c644b1b36f4d19154490e0/users/b970f9cb8fe444f3a25fac913f23f84d/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:56:37.198268 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311b5454-5e0b-4b1d-97b1-8035258a6195 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:37.198555 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311b5454-5e0b-4b1d-97b1-8035258a6195 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:37.198822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311b5454-5e0b-4b1d-97b1-8035258a6195 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:37.227615 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-311b5454-5e0b-4b1d-97b1-8035258a6195 None None] MFA Rules not processed for user `b970f9cb8fe444f3a25fac913f23f84d`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:37.228821 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-311b5454-5e0b-4b1d-97b1-8035258a6195 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:37.242323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:37.284140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1296/2594] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:37 2022] POST /identity/v3/users => generated 302 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:56:37.288323 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251667aa-7406-4043-80aa-f785580c8425 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:37.299167 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251667aa-7406-4043-80aa-f785580c8425 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:37.308259 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1299/2595] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:37 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:56:37.318422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311b5454-5e0b-4b1d-97b1-8035258a6195 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:37.319856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-311b5454-5e0b-4b1d-97b1-8035258a6195 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:37.384325 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5771a4-a92e-4591-a763-3f173b7169a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:37.384933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5771a4-a92e-4591-a763-3f173b7169a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:37.385216 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5771a4-a92e-4591-a763-3f173b7169a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:37.385341 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:37.385516 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5771a4-a92e-4591-a763-3f173b7169a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:37.386134 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5771a4-a92e-4591-a763-3f173b7169a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:37.387946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5771a4-a92e-4591-a763-3f173b7169a3 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:37.389998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311b5454-5e0b-4b1d-97b1-8035258a6195 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:37.394133 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb5d35b-53f6-4377-86a9-770324a6da54 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ceb9fc31c345ccabaee8f30a3982a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:37.395323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb5d35b-53f6-4377-86a9-770324a6da54 service glance] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:37.395655 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb5d35b-53f6-4377-86a9-770324a6da54 service glance] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:37.395867 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb5d35b-53f6-4377-86a9-770324a6da54 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:37.402699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1297/2596] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:37 2022] GET /identity/v3/roles => generated 1420 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:37.403701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5771a4-a92e-4591-a763-3f173b7169a3 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:37.405643 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-5d5771a4-a92e-4591-a763-3f173b7169a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:37.406326 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb5d35b-53f6-4377-86a9-770324a6da54 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:37.407808 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb5d35b-53f6-4377-86a9-770324a6da54 service glance] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:37.417710 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1300/2597] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:37 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:37.417932 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5771a4-a92e-4591-a763-3f173b7169a3 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:37.419022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb5d35b-53f6-4377-86a9-770324a6da54 service glance] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:37.423541 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5206f6-524a-4aae-ba4a-49ef7737114e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:37.424174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5206f6-524a-4aae-ba4a-49ef7737114e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:37.428688 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5206f6-524a-4aae-ba4a-49ef7737114e admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:37.428688 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5206f6-524a-4aae-ba4a-49ef7737114e admin admin] PATH_INFO: `/v3/projects/2d777a6283fd49d0868014430654788b` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:37.428688 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5206f6-524a-4aae-ba4a-49ef7737114e admin admin] RBAC: Authorizing `identity:delete_project(project_id=2d777a6283fd49d0868014430654788b)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:37.428688 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5206f6-524a-4aae-ba4a-49ef7737114e admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:37.431776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb5d35b-53f6-4377-86a9-770324a6da54 service glance] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:37.439866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a24d3fe-cc1a-4a12-b51a-7fd8e17a1c34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:37.440980 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a24d3fe-cc1a-4a12-b51a-7fd8e17a1c34 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:37.441300 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a24d3fe-cc1a-4a12-b51a-7fd8e17a1c34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:37.441665 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a24d3fe-cc1a-4a12-b51a-7fd8e17a1c34 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:37.442883 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a24d3fe-cc1a-4a12-b51a-7fd8e17a1c34 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:37.444940 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a24d3fe-cc1a-4a12-b51a-7fd8e17a1c34 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:37.466431 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1301/2598] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:37 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:37.470259 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a24d3fe-cc1a-4a12-b51a-7fd8e17a1c34 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:37.480483 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a24d3fe-cc1a-4a12-b51a-7fd8e17a1c34 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:37.485471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb872f5-be12-4502-9b1e-3ce000dbf415 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:37.486194 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb872f5-be12-4502-9b1e-3ce000dbf415 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:37.486531 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb872f5-be12-4502-9b1e-3ce000dbf415 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:37.487760 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb872f5-be12-4502-9b1e-3ce000dbf415 admin admin] PATH_INFO: `/v3/projects/3a898b286911461a8a0ac89deeca754f/users/929be71609904c1488e131a7a39a62a6/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:37.504138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb872f5-be12-4502-9b1e-3ce000dbf415 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a898b286911461a8a0ac89deeca754f, user_id=929be71609904c1488e131a7a39a62a6, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:37.506856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb872f5-be12-4502-9b1e-3ce000dbf415 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:37.629834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1302/2599] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:37 2022] PUT /identity/v3/projects/3a898b286911461a8a0ac89deeca754f/users/929be71609904c1488e131a7a39a62a6/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:37.636714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5719ecf7-b60f-4175-8944-d731f4afd697 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:37.636847 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5719ecf7-b60f-4175-8944-d731f4afd697 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:37.637149 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5719ecf7-b60f-4175-8944-d731f4afd697 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:37.650681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5d5206f6-524a-4aae-ba4a-49ef7737114e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2d777a6283fd49d0868014430654788b', 'request_id': 'req-5d5206f6-524a-4aae-ba4a-49ef7737114e'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:37.661898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-5719ecf7-b60f-4175-8944-d731f4afd697 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:37.663419 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5719ecf7-b60f-4175-8944-d731f4afd697 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:37.735394 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1303/2600] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:37.745150 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5719ecf7-b60f-4175-8944-d731f4afd697 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:37.745637 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5d5206f6-524a-4aae-ba4a-49ef7737114e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2d777a6283fd49d0868014430654788b', 'request_id': 'req-5d5206f6-524a-4aae-ba4a-49ef7737114e'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:37.746758 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5719ecf7-b60f-4175-8944-d731f4afd697 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:37.754783 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5d5206f6-524a-4aae-ba4a-49ef7737114e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2d777a6283fd49d0868014430654788b', 'request_id': 'req-5d5206f6-524a-4aae-ba4a-49ef7737114e'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:37.771668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5d5206f6-524a-4aae-ba4a-49ef7737114e admin admin] The token cache is being invalidate because project 2d777a6283fd49d0868014430654788b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:37.772054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5d5206f6-524a-4aae-ba4a-49ef7737114e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:37.772912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5d5206f6-524a-4aae-ba4a-49ef7737114e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:37.774572 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1298/2601] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:37 2022] DELETE /identity/v3/projects/2d777a6283fd49d0868014430654788b => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:37.780885 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e50e0fb-9bea-4365-aaff-10d3f4af0cf0 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:37.781973 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e50e0fb-9bea-4365-aaff-10d3f4af0cf0 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:37.783646 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:37.784304 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e50e0fb-9bea-4365-aaff-10d3f4af0cf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:37.805822 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-9e50e0fb-9bea-4365-aaff-10d3f4af0cf0 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:37.806700 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-9e50e0fb-9bea-4365-aaff-10d3f4af0cf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:37.815742 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1299/2602] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:37.823562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3118b38-0205-4d2f-a7e3-7ac9621b8e21 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:37.825331 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3118b38-0205-4d2f-a7e3-7ac9621b8e21 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:37.825331 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3118b38-0205-4d2f-a7e3-7ac9621b8e21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:37.831112 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5719ecf7-b60f-4175-8944-d731f4afd697 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:37.834944 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97ce1f7-ca33-4866-b168-578777225238 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:37.835568 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97ce1f7-ca33-4866-b168-578777225238 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:37.835852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97ce1f7-ca33-4866-b168-578777225238 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:37.836131 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97ce1f7-ca33-4866-b168-578777225238 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:37.837869 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d97ce1f7-ca33-4866-b168-578777225238 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:37.843904 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-d3118b38-0205-4d2f-a7e3-7ac9621b8e21 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:37.845072 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d3118b38-0205-4d2f-a7e3-7ac9621b8e21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:37.850461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97ce1f7-ca33-4866-b168-578777225238 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:37.851922 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97ce1f7-ca33-4866-b168-578777225238 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:37.856478 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1300/2603] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:56:37.860714 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3118b38-0205-4d2f-a7e3-7ac9621b8e21 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:37.865507 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1304/2604] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:37.869784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3118b38-0205-4d2f-a7e3-7ac9621b8e21 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:37.874935 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c16346-ca05-46e2-b2d6-78cac3ca9ec0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:37.875556 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c16346-ca05-46e2-b2d6-78cac3ca9ec0 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:37.875840 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c16346-ca05-46e2-b2d6-78cac3ca9ec0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:37.876111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c16346-ca05-46e2-b2d6-78cac3ca9ec0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:37.876762 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c16346-ca05-46e2-b2d6-78cac3ca9ec0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:56:37.877122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c16346-ca05-46e2-b2d6-78cac3ca9ec0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:37.878797 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c16346-ca05-46e2-b2d6-78cac3ca9ec0 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:37.882446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:37.892553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1301/2605] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:56:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:56:37.896215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97ce1f7-ca33-4866-b168-578777225238 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:37.905949 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97ce1f7-ca33-4866-b168-578777225238 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:37.910686 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb48cab-70c9-405d-9ce5-6b33fa956b70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:37.911328 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb48cab-70c9-405d-9ce5-6b33fa956b70 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:37.911614 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb48cab-70c9-405d-9ce5-6b33fa956b70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:37.911895 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb48cab-70c9-405d-9ce5-6b33fa956b70 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:37.912522 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb48cab-70c9-405d-9ce5-6b33fa956b70 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:37.914414 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb48cab-70c9-405d-9ce5-6b33fa956b70 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:38.007249 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1305/2606] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:37 2022] POST /identity/v3/projects => generated 375 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:38.008493 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c16346-ca05-46e2-b2d6-78cac3ca9ec0 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:38.018880 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c16346-ca05-46e2-b2d6-78cac3ca9ec0 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:38.080282 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab73273-9e14-4179-be66-1716367c1c7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:38.080994 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab73273-9e14-4179-be66-1716367c1c7d admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:38.081380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab73273-9e14-4179-be66-1716367c1c7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:38.081716 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab73273-9e14-4179-be66-1716367c1c7d admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:38.082453 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab73273-9e14-4179-be66-1716367c1c7d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:38.084662 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab73273-9e14-4179-be66-1716367c1c7d admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:38.196984 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1302/2607] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:38 2022] POST /identity/v3/users => generated 288 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:38.201538 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb48cab-70c9-405d-9ce5-6b33fa956b70 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:38.215058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb48cab-70c9-405d-9ce5-6b33fa956b70 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:38.222991 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cc74bf-4494-4cbb-83af-f355ef98f6af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:38.224333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cc74bf-4494-4cbb-83af-f355ef98f6af admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:38.225120 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cc74bf-4494-4cbb-83af-f355ef98f6af admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:38.225655 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cc74bf-4494-4cbb-83af-f355ef98f6af admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:38.226751 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cc74bf-4494-4cbb-83af-f355ef98f6af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:38.230078 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cc74bf-4494-4cbb-83af-f355ef98f6af admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:38.245257 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1306/2608] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:38 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:38.251393 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab73273-9e14-4179-be66-1716367c1c7d admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:38.261712 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab73273-9e14-4179-be66-1716367c1c7d admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:38.266843 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50457445-45ee-41b2-be32-833efdf75980 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:38.267500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50457445-45ee-41b2-be32-833efdf75980 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:38.267781 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50457445-45ee-41b2-be32-833efdf75980 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:38.268056 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50457445-45ee-41b2-be32-833efdf75980 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:38.268701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50457445-45ee-41b2-be32-833efdf75980 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:38.270417 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50457445-45ee-41b2-be32-833efdf75980 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:38.282956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1303/2609] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:38 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:38.288064 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cc74bf-4494-4cbb-83af-f355ef98f6af admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:38.299584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cc74bf-4494-4cbb-83af-f355ef98f6af admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:38.304543 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1e9ee9-8d54-415a-9248-79d6007fc041 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:38.307364 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1e9ee9-8d54-415a-9248-79d6007fc041 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:38.307364 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1e9ee9-8d54-415a-9248-79d6007fc041 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:38.307364 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1e9ee9-8d54-415a-9248-79d6007fc041 admin admin] PATH_INFO: `/v3/projects/404f951c8fa54b81a3692b541c47981c/users/411d8a02f2854e709beecf7a6a06dda0/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:38.322885 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1e9ee9-8d54-415a-9248-79d6007fc041 admin admin] RBAC: Authorizing `identity:create_grant(project_id=404f951c8fa54b81a3692b541c47981c, user_id=411d8a02f2854e709beecf7a6a06dda0, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:38.325549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1e9ee9-8d54-415a-9248-79d6007fc041 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:38.432277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1307/2610] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:38 2022] PUT /identity/v3/projects/404f951c8fa54b81a3692b541c47981c/users/411d8a02f2854e709beecf7a6a06dda0/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:38.443405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2a9cfd-6bc6-42e1-ac3e-65f61e16c3c5 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:38.445133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2a9cfd-6bc6-42e1-ac3e-65f61e16c3c5 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:38.445133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2a9cfd-6bc6-42e1-ac3e-65f61e16c3c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:38.492523 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-0a2a9cfd-6bc6-42e1-ac3e-65f61e16c3c5 None None] MFA Rules not processed for user `411d8a02f2854e709beecf7a6a06dda0`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:38.493797 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-0a2a9cfd-6bc6-42e1-ac3e-65f61e16c3c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:38.591178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1304/2611] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:38 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:38.595997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1e9ee9-8d54-415a-9248-79d6007fc041 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:38.669451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1e9ee9-8d54-415a-9248-79d6007fc041 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:38.675084 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a36c469-0539-4af3-beef-9ea6c627bb26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:38.675084 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a36c469-0539-4af3-beef-9ea6c627bb26 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:38.675408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a36c469-0539-4af3-beef-9ea6c627bb26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:38.675584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a36c469-0539-4af3-beef-9ea6c627bb26 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:38.676214 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a36c469-0539-4af3-beef-9ea6c627bb26 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:38.678343 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a36c469-0539-4af3-beef-9ea6c627bb26 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:38.763214 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:38.802376 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1308/2612] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:38 2022] POST /identity/v3/projects => generated 377 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:38.803583 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2a9cfd-6bc6-42e1-ac3e-65f61e16c3c5 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:38.815803 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2a9cfd-6bc6-42e1-ac3e-65f61e16c3c5 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:38.900366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6967736-0c87-43f8-a44e-e88777aee9d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:38.901067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6967736-0c87-43f8-a44e-e88777aee9d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:38.901344 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6967736-0c87-43f8-a44e-e88777aee9d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:38.906322 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6967736-0c87-43f8-a44e-e88777aee9d6 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:38.906322 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6967736-0c87-43f8-a44e-e88777aee9d6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:38.906322 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6967736-0c87-43f8-a44e-e88777aee9d6 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:38.909782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:39.014822 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1305/2613] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:38 2022] POST /identity/v3/users => generated 289 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:39.016085 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a36c469-0539-4af3-beef-9ea6c627bb26 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:39.028311 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a36c469-0539-4af3-beef-9ea6c627bb26 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:39.033829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9184787b-5281-4a61-acd3-a0b42f2d9cf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:39.034581 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9184787b-5281-4a61-acd3-a0b42f2d9cf7 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:39.038371 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9184787b-5281-4a61-acd3-a0b42f2d9cf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:39.038371 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9184787b-5281-4a61-acd3-a0b42f2d9cf7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:39.038371 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9184787b-5281-4a61-acd3-a0b42f2d9cf7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:39.038844 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9184787b-5281-4a61-acd3-a0b42f2d9cf7 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:39.054362 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1309/2614] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:39 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:39.056720 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6967736-0c87-43f8-a44e-e88777aee9d6 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:39.069324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6967736-0c87-43f8-a44e-e88777aee9d6 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:39.074450 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028b1ee4-e757-41fd-803f-3dd041307c5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:39.075975 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028b1ee4-e757-41fd-803f-3dd041307c5a admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:39.076350 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028b1ee4-e757-41fd-803f-3dd041307c5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:39.076735 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028b1ee4-e757-41fd-803f-3dd041307c5a admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:39.077494 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028b1ee4-e757-41fd-803f-3dd041307c5a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:39.079579 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028b1ee4-e757-41fd-803f-3dd041307c5a admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:39.096732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1306/2615] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:39 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:39.100661 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9184787b-5281-4a61-acd3-a0b42f2d9cf7 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:39.111256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9184787b-5281-4a61-acd3-a0b42f2d9cf7 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:39.117849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241e0c73-9b2e-4150-8ade-798f59184d8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:39.118544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241e0c73-9b2e-4150-8ade-798f59184d8f admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:39.118823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241e0c73-9b2e-4150-8ade-798f59184d8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:39.119137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241e0c73-9b2e-4150-8ade-798f59184d8f admin admin] PATH_INFO: `/v3/projects/217bb031aa714765866208831eba725e/users/9351d09333d7416d8e5b559e23217f80/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:39.134782 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241e0c73-9b2e-4150-8ade-798f59184d8f admin admin] RBAC: Authorizing `identity:create_grant(project_id=217bb031aa714765866208831eba725e, user_id=9351d09333d7416d8e5b559e23217f80, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:39.138823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241e0c73-9b2e-4150-8ade-798f59184d8f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:39.220959 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1310/2616] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:39 2022] PUT /identity/v3/projects/217bb031aa714765866208831eba725e/users/9351d09333d7416d8e5b559e23217f80/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:39.232500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68da203f-a0f2-43c9-92be-ad5b7d9a02bc None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:39.232500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68da203f-a0f2-43c9-92be-ad5b7d9a02bc None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:39.232500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68da203f-a0f2-43c9-92be-ad5b7d9a02bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:39.265033 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-68da203f-a0f2-43c9-92be-ad5b7d9a02bc None None] MFA Rules not processed for user `9351d09333d7416d8e5b559e23217f80`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:39.266156 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-68da203f-a0f2-43c9-92be-ad5b7d9a02bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:39.344399 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1307/2617] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:39 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:39.357175 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241e0c73-9b2e-4150-8ade-798f59184d8f admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:39.410416 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579757e5-c00f-4c2e-9b2f-d43f1ccf2568 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:39.410833 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579757e5-c00f-4c2e-9b2f-d43f1ccf2568 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:39.411237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579757e5-c00f-4c2e-9b2f-d43f1ccf2568 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:39.429901 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:39.437487 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241e0c73-9b2e-4150-8ade-798f59184d8f admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:39.443847 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae24a14-4f88-4166-b391-b367740d7b5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:39.444483 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae24a14-4f88-4166-b391-b367740d7b5d admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:39.444762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae24a14-4f88-4166-b391-b367740d7b5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:39.445040 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae24a14-4f88-4166-b391-b367740d7b5d admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:39.445680 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae24a14-4f88-4166-b391-b367740d7b5d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:39.448757 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-579757e5-c00f-4c2e-9b2f-d43f1ccf2568 None None] MFA Rules not processed for user `929be71609904c1488e131a7a39a62a6`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:39.450208 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-579757e5-c00f-4c2e-9b2f-d43f1ccf2568 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:39.450359 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae24a14-4f88-4166-b391-b367740d7b5d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:39.556145 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1311/2618] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:39 2022] POST /identity/v3/projects => generated 377 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:56:39.562450 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae24a14-4f88-4166-b391-b367740d7b5d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:39.648124 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae24a14-4f88-4166-b391-b367740d7b5d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:39.657555 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffdf800-49cc-4f32-8ca9-90e18be2716f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:39.658186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffdf800-49cc-4f32-8ca9-90e18be2716f admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:39.658463 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffdf800-49cc-4f32-8ca9-90e18be2716f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:39.658928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffdf800-49cc-4f32-8ca9-90e18be2716f admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:39.659689 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ffdf800-49cc-4f32-8ca9-90e18be2716f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:39.662025 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ffdf800-49cc-4f32-8ca9-90e18be2716f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:39.683210 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1308/2619] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:39 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 275 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:56:39.688050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579757e5-c00f-4c2e-9b2f-d43f1ccf2568 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:39.689886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-579757e5-c00f-4c2e-9b2f-d43f1ccf2568 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:39.705905 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579757e5-c00f-4c2e-9b2f-d43f1ccf2568 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:39.710583 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac61906-7275-454d-81cf-f579622534e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:39.711384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac61906-7275-454d-81cf-f579622534e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:39.711733 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac61906-7275-454d-81cf-f579622534e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:39.712029 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac61906-7275-454d-81cf-f579622534e2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:39.712668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac61906-7275-454d-81cf-f579622534e2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:39.714793 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac61906-7275-454d-81cf-f579622534e2 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:39.826317 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1312/2620] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:39 2022] POST /identity/v3/users => generated 295 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:56:39.830707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffdf800-49cc-4f32-8ca9-90e18be2716f admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:39.844603 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffdf800-49cc-4f32-8ca9-90e18be2716f admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:39.849151 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1309/2621] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:39 2022] POST /identity/v3/projects => generated 405 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:56:39.852856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c941eea-49aa-4cc8-9e32-dc62fef60a3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:39.852856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c941eea-49aa-4cc8-9e32-dc62fef60a3f admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:39.852856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c941eea-49aa-4cc8-9e32-dc62fef60a3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:39.852856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c941eea-49aa-4cc8-9e32-dc62fef60a3f admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:39.853320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c941eea-49aa-4cc8-9e32-dc62fef60a3f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:39.854674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac61906-7275-454d-81cf-f579622534e2 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:39.854939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c941eea-49aa-4cc8-9e32-dc62fef60a3f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:39.867391 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1313/2622] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:39 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:56:39.874698 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c941eea-49aa-4cc8-9e32-dc62fef60a3f admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:39.891804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c941eea-49aa-4cc8-9e32-dc62fef60a3f admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:39.935871 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac61906-7275-454d-81cf-f579622534e2 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:39.942225 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ea4295-bc24-4ec5-91f1-4a2f644a04ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:39.943143 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ea4295-bc24-4ec5-91f1-4a2f644a04ee admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:39.943493 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ea4295-bc24-4ec5-91f1-4a2f644a04ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:39.943878 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ea4295-bc24-4ec5-91f1-4a2f644a04ee admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:39.944749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ea4295-bc24-4ec5-91f1-4a2f644a04ee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:39.947037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ea4295-bc24-4ec5-91f1-4a2f644a04ee admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:39.953937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:39.972664 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd141364-e83b-43e0-a162-fab68eecf716 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:39.973405 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd141364-e83b-43e0-a162-fab68eecf716 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:39.973752 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd141364-e83b-43e0-a162-fab68eecf716 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:39.974116 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd141364-e83b-43e0-a162-fab68eecf716 admin admin] PATH_INFO: `/v3/projects/08a4fc4962d144ffa5d50f8387431718/users/31a36f2b13f54341baf4ce3865d63f0d/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:39.991439 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd141364-e83b-43e0-a162-fab68eecf716 admin admin] RBAC: Authorizing `identity:create_grant(project_id=08a4fc4962d144ffa5d50f8387431718, user_id=31a36f2b13f54341baf4ce3865d63f0d, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:39.994244 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd141364-e83b-43e0-a162-fab68eecf716 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:40.089020 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1310/2623] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:39 2022] POST /identity/v3/users => generated 309 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:40.093037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ea4295-bc24-4ec5-91f1-4a2f644a04ee admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:40.103269 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ea4295-bc24-4ec5-91f1-4a2f644a04ee admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:40.109171 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3d763a-1392-4c3b-8de1-6331e7dc5ca8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:40.109758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3d763a-1392-4c3b-8de1-6331e7dc5ca8 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:40.110091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3d763a-1392-4c3b-8de1-6331e7dc5ca8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:40.110361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3d763a-1392-4c3b-8de1-6331e7dc5ca8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:40.111032 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3d763a-1392-4c3b-8de1-6331e7dc5ca8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:40.112636 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3d763a-1392-4c3b-8de1-6331e7dc5ca8 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:40.123956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1311/2624] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:40 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:56:40.127845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3d763a-1392-4c3b-8de1-6331e7dc5ca8 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:40.139883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3d763a-1392-4c3b-8de1-6331e7dc5ca8 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:40.141881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1314/2625] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:39 2022] PUT /identity/v3/projects/08a4fc4962d144ffa5d50f8387431718/users/31a36f2b13f54341baf4ce3865d63f0d/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:40.151249 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd141364-e83b-43e0-a162-fab68eecf716 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:40.212779 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd25c85-db02-4919-8911-ed04e84e985b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:40.213389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd25c85-db02-4919-8911-ed04e84e985b admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:40.213674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd25c85-db02-4919-8911-ed04e84e985b admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:40.213946 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd25c85-db02-4919-8911-ed04e84e985b admin admin] PATH_INFO: `/v3/projects/3057271a87e04c639ceaf0657d0c6625/users/d9d31ba104844ad085e9cfa5d3f600b0/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:40.232021 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd141364-e83b-43e0-a162-fab68eecf716 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:40.239103 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd25c85-db02-4919-8911-ed04e84e985b admin admin] RBAC: Authorizing `identity:create_grant(project_id=3057271a87e04c639ceaf0657d0c6625, user_id=d9d31ba104844ad085e9cfa5d3f600b0, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:40.241536 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd25c85-db02-4919-8911-ed04e84e985b admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:40.243244 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:40.246004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c95997-9223-49e7-83e8-a2a78c5822a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:40.246733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c95997-9223-49e7-83e8-a2a78c5822a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:40.247147 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c95997-9223-49e7-83e8-a2a78c5822a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:40.247354 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c95997-9223-49e7-83e8-a2a78c5822a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:40.257189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c95997-9223-49e7-83e8-a2a78c5822a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:40.257189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c95997-9223-49e7-83e8-a2a78c5822a3 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:40.260829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1315/2626] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:40 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:56:40.265221 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c95997-9223-49e7-83e8-a2a78c5822a3 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:40.275394 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c95997-9223-49e7-83e8-a2a78c5822a3 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:40.286545 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5adf4f54-98b9-40f4-8fbe-5447eff33c4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:40.287216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adf4f54-98b9-40f4-8fbe-5447eff33c4a admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:40.287505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adf4f54-98b9-40f4-8fbe-5447eff33c4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:40.287904 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adf4f54-98b9-40f4-8fbe-5447eff33c4a admin admin] PATH_INFO: `/v3/projects/08a4fc4962d144ffa5d50f8387431718/users/31a36f2b13f54341baf4ce3865d63f0d/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:40.289910 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5adf4f54-98b9-40f4-8fbe-5447eff33c4a admin admin] RBAC: Authorizing `identity:create_grant(project_id=08a4fc4962d144ffa5d50f8387431718, user_id=31a36f2b13f54341baf4ce3865d63f0d, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:40.293582 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5adf4f54-98b9-40f4-8fbe-5447eff33c4a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:40.307010 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1316/2627] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:40 2022] PUT /identity/v3/projects/08a4fc4962d144ffa5d50f8387431718/users/31a36f2b13f54341baf4ce3865d63f0d/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:56:40.318624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c5055f-3359-4190-8cd4-688c6281390c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:40.319072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c5055f-3359-4190-8cd4-688c6281390c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:40.319451 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c5055f-3359-4190-8cd4-688c6281390c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:40.359305 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-d8c5055f-3359-4190-8cd4-688c6281390c None None] MFA Rules not processed for user `31a36f2b13f54341baf4ce3865d63f0d`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:40.361169 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d8c5055f-3359-4190-8cd4-688c6281390c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:40.379192 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1312/2628] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:40 2022] PUT /identity/v3/projects/3057271a87e04c639ceaf0657d0c6625/users/d9d31ba104844ad085e9cfa5d3f600b0/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:56:40.398469 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd25c85-db02-4919-8911-ed04e84e985b admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:40.448422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1317/2629] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:40 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:56:40.462578 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c5055f-3359-4190-8cd4-688c6281390c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:40.464051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d8c5055f-3359-4190-8cd4-688c6281390c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:40.469336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd25c85-db02-4919-8911-ed04e84e985b admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:40.474171 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301f3bec-2c20-4e79-be3a-6bc2be340a74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:40.474850 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301f3bec-2c20-4e79-be3a-6bc2be340a74 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:40.475120 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301f3bec-2c20-4e79-be3a-6bc2be340a74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:40.486857 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301f3bec-2c20-4e79-be3a-6bc2be340a74 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:40.486857 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301f3bec-2c20-4e79-be3a-6bc2be340a74 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:40.486857 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301f3bec-2c20-4e79-be3a-6bc2be340a74 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:40.488156 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1313/2630] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:40 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:56:40.492242 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301f3bec-2c20-4e79-be3a-6bc2be340a74 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:40.505685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301f3bec-2c20-4e79-be3a-6bc2be340a74 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:40.515709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6ff5df-246f-4192-9cd9-831bc689e2d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:40.515709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6ff5df-246f-4192-9cd9-831bc689e2d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:40.515709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6ff5df-246f-4192-9cd9-831bc689e2d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:40.515709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6ff5df-246f-4192-9cd9-831bc689e2d6 admin admin] PATH_INFO: `/v3/projects/3057271a87e04c639ceaf0657d0c6625/users/d9d31ba104844ad085e9cfa5d3f600b0/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:40.516362 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6ff5df-246f-4192-9cd9-831bc689e2d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3057271a87e04c639ceaf0657d0c6625, user_id=d9d31ba104844ad085e9cfa5d3f600b0, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:40.518503 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6ff5df-246f-4192-9cd9-831bc689e2d6 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:40.532920 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1314/2631] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:40 2022] PUT /identity/v3/projects/3057271a87e04c639ceaf0657d0c6625/users/d9d31ba104844ad085e9cfa5d3f600b0/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:56:40.539021 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c5055f-3359-4190-8cd4-688c6281390c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:40.592219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccf8467-d054-4736-8121-cfcd337983bc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:40.592770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccf8467-d054-4736-8121-cfcd337983bc service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:40.593046 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccf8467-d054-4736-8121-cfcd337983bc service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:40.593316 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccf8467-d054-4736-8121-cfcd337983bc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:40.654068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccf8467-d054-4736-8121-cfcd337983bc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:40.655595 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccf8467-d054-4736-8121-cfcd337983bc service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:40.664617 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1318/2632] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:40 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:56:40.688787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:40.757850 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:42.130788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9072cc-1821-4a60-8963-66d89876f2e3 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:42.131379 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9072cc-1821-4a60-8963-66d89876f2e3 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:42.131379 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9072cc-1821-4a60-8963-66d89876f2e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:42.160357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-5c9072cc-1821-4a60-8963-66d89876f2e3 None None] MFA Rules not processed for user `d9d31ba104844ad085e9cfa5d3f600b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:42.161868 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-5c9072cc-1821-4a60-8963-66d89876f2e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:42.243997 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1315/2633] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:42 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:56:42.782536 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7f8cd2-e246-4e78-bcdc-f61688a4b628 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:42.783116 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7f8cd2-e246-4e78-bcdc-f61688a4b628 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:42.783518 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7f8cd2-e246-4e78-bcdc-f61688a4b628 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:42.816573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-6b7f8cd2-e246-4e78-bcdc-f61688a4b628 None None] MFA Rules not processed for user `d40810f61ead4c3da414194b325da80a`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:42.818852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6b7f8cd2-e246-4e78-bcdc-f61688a4b628 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:42.895361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1319/2634] 10.222.0.49 () {58 vars in 967 bytes} [Fri Sep 16 03:56:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:42.896890 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9072cc-1821-4a60-8963-66d89876f2e3 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:42.908414 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9072cc-1821-4a60-8963-66d89876f2e3 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:42.915325 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f11b2d-d107-40dc-a39a-7c1edcd6685a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:42.915476 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f11b2d-d107-40dc-a39a-7c1edcd6685a service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:42.915892 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f11b2d-d107-40dc-a39a-7c1edcd6685a service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:42.916085 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f11b2d-d107-40dc-a39a-7c1edcd6685a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:42.927971 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:42.983146 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f11b2d-d107-40dc-a39a-7c1edcd6685a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:42.984601 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f11b2d-d107-40dc-a39a-7c1edcd6685a service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:42.994232 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1316/2635] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:42 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:56:43.173293 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7f8cd2-e246-4e78-bcdc-f61688a4b628 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:43.175090 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6b7f8cd2-e246-4e78-bcdc-f61688a4b628 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:43.252815 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7f8cd2-e246-4e78-bcdc-f61688a4b628 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:43.257894 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4179c9-c341-485d-88d6-e254c946ad5d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa30da9972e140d7905c522576681429', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:43.258511 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4179c9-c341-485d-88d6-e254c946ad5d service swift] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:43.258789 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4179c9-c341-485d-88d6-e254c946ad5d service swift] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:43.259096 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4179c9-c341-485d-88d6-e254c946ad5d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:43.268801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4179c9-c341-485d-88d6-e254c946ad5d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:43.270380 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4179c9-c341-485d-88d6-e254c946ad5d service swift] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:43.278565 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1320/2636] 10.222.0.49 () {60 vars in 1384 bytes} [Fri Sep 16 03:56:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 16 03:56:44.113425 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f11b2d-d107-40dc-a39a-7c1edcd6685a service cinder] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:44.178073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f11b2d-d107-40dc-a39a-7c1edcd6685a service cinder] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:44.183267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8e13be-2962-4694-bfb7-db8336f516c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:44.185127 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8e13be-2962-4694-bfb7-db8336f516c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:44.185622 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8e13be-2962-4694-bfb7-db8336f516c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:44.185949 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8e13be-2962-4694-bfb7-db8336f516c3 admin admin] PATH_INFO: `/v3/users/929be71609904c1488e131a7a39a62a6` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:44.187124 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8e13be-2962-4694-bfb7-db8336f516c3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=929be71609904c1488e131a7a39a62a6)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:44.189031 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8e13be-2962-4694-bfb7-db8336f516c3 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:44.486126 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-2d8e13be-2962-4694-bfb7-db8336f516c3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '929be71609904c1488e131a7a39a62a6'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:44.699011 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-2d8e13be-2962-4694-bfb7-db8336f516c3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '929be71609904c1488e131a7a39a62a6'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:44.715827 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-2d8e13be-2962-4694-bfb7-db8336f516c3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '929be71609904c1488e131a7a39a62a6'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:44.716947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-2d8e13be-2962-4694-bfb7-db8336f516c3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '929be71609904c1488e131a7a39a62a6'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:44.724589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-2d8e13be-2962-4694-bfb7-db8336f516c3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '929be71609904c1488e131a7a39a62a6'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:44.730813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1317/2637] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:44 2022] DELETE /identity/v3/users/929be71609904c1488e131a7a39a62a6 => generated 0 bytes in 618 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:45.015202 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4179c9-c341-485d-88d6-e254c946ad5d service swift] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:45.025827 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4179c9-c341-485d-88d6-e254c946ad5d service swift] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:45.098723 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:45.106148 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6425550b-3904-4ba7-aa64-46936c467ad2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:45.107183 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6425550b-3904-4ba7-aa64-46936c467ad2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:45.107647 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6425550b-3904-4ba7-aa64-46936c467ad2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:45.108142 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6425550b-3904-4ba7-aa64-46936c467ad2 admin admin] PATH_INFO: `/v3/projects/3a898b286911461a8a0ac89deeca754f` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:45.109604 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6425550b-3904-4ba7-aa64-46936c467ad2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3a898b286911461a8a0ac89deeca754f)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:45.111737 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6425550b-3904-4ba7-aa64-46936c467ad2 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:45.251734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8e13be-2962-4694-bfb7-db8336f516c3 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:45.252263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6425550b-3904-4ba7-aa64-46936c467ad2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3a898b286911461a8a0ac89deeca754f', 'request_id': 'req-6425550b-3904-4ba7-aa64-46936c467ad2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:45.253526 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-2d8e13be-2962-4694-bfb7-db8336f516c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:45.261774 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6425550b-3904-4ba7-aa64-46936c467ad2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3a898b286911461a8a0ac89deeca754f', 'request_id': 'req-6425550b-3904-4ba7-aa64-46936c467ad2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:45.294369 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:45.321361 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8e13be-2962-4694-bfb7-db8336f516c3 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:45.377002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a09035-d7dc-4bb7-8783-3fa7e6c6fd4b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:45.379318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6425550b-3904-4ba7-aa64-46936c467ad2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3a898b286911461a8a0ac89deeca754f', 'request_id': 'req-6425550b-3904-4ba7-aa64-46936c467ad2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:45.381550 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a09035-d7dc-4bb7-8783-3fa7e6c6fd4b service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:45.381822 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a09035-d7dc-4bb7-8783-3fa7e6c6fd4b service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:45.382232 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a09035-d7dc-4bb7-8783-3fa7e6c6fd4b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:45.384672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e9a09035-d7dc-4bb7-8783-3fa7e6c6fd4b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:45.393714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6425550b-3904-4ba7-aa64-46936c467ad2 admin admin] The token cache is being invalidate because project 3a898b286911461a8a0ac89deeca754f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:45.394298 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6425550b-3904-4ba7-aa64-46936c467ad2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:45.395355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6425550b-3904-4ba7-aa64-46936c467ad2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:45.400193 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1321/2638] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:45 2022] DELETE /identity/v3/projects/3a898b286911461a8a0ac89deeca754f => generated 0 bytes in 382 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:45.404360 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6425550b-3904-4ba7-aa64-46936c467ad2 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:45.404360 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6425550b-3904-4ba7-aa64-46936c467ad2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:45.461086 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a09035-d7dc-4bb7-8783-3fa7e6c6fd4b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:45.463167 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a09035-d7dc-4bb7-8783-3fa7e6c6fd4b service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:45.474161 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6425550b-3904-4ba7-aa64-46936c467ad2 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:45.474988 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1318/2639] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:45 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:45.482030 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520a35be-b8a5-46d8-80ba-66883496cec1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:45.482552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520a35be-b8a5-46d8-80ba-66883496cec1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:45.483161 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520a35be-b8a5-46d8-80ba-66883496cec1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:45.483497 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520a35be-b8a5-46d8-80ba-66883496cec1 admin admin] PATH_INFO: `/v3/users/d9d31ba104844ad085e9cfa5d3f600b0` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:45.484671 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520a35be-b8a5-46d8-80ba-66883496cec1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d9d31ba104844ad085e9cfa5d3f600b0)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:45.486961 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520a35be-b8a5-46d8-80ba-66883496cec1 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:45.729104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-520a35be-b8a5-46d8-80ba-66883496cec1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd9d31ba104844ad085e9cfa5d3f600b0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:45.745775 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-520a35be-b8a5-46d8-80ba-66883496cec1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd9d31ba104844ad085e9cfa5d3f600b0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:45.836351 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-520a35be-b8a5-46d8-80ba-66883496cec1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd9d31ba104844ad085e9cfa5d3f600b0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:45.837002 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-520a35be-b8a5-46d8-80ba-66883496cec1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd9d31ba104844ad085e9cfa5d3f600b0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:45.845419 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-520a35be-b8a5-46d8-80ba-66883496cec1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd9d31ba104844ad085e9cfa5d3f600b0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:45.851612 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1322/2640] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:45 2022] DELETE /identity/v3/users/d9d31ba104844ad085e9cfa5d3f600b0 => generated 0 bytes in 451 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:46.390394 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a09035-d7dc-4bb7-8783-3fa7e6c6fd4b service cinder] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:46.392397 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e9a09035-d7dc-4bb7-8783-3fa7e6c6fd4b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:46.406764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a09035-d7dc-4bb7-8783-3fa7e6c6fd4b service cinder] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:46.478692 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2aadc06-84b2-4209-913d-c5c78fde951d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:46.479950 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aadc06-84b2-4209-913d-c5c78fde951d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:46.480238 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aadc06-84b2-4209-913d-c5c78fde951d admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:46.480516 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aadc06-84b2-4209-913d-c5c78fde951d admin admin] PATH_INFO: `/v3/projects/3057271a87e04c639ceaf0657d0c6625` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:46.482362 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2aadc06-84b2-4209-913d-c5c78fde951d admin admin] RBAC: Authorizing `identity:delete_project(project_id=3057271a87e04c639ceaf0657d0c6625)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:46.484294 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2aadc06-84b2-4209-913d-c5c78fde951d admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:46.627379 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:46.634444 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f2aadc06-84b2-4209-913d-c5c78fde951d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3057271a87e04c639ceaf0657d0c6625', 'request_id': 'req-f2aadc06-84b2-4209-913d-c5c78fde951d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:46.727825 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f2aadc06-84b2-4209-913d-c5c78fde951d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3057271a87e04c639ceaf0657d0c6625', 'request_id': 'req-f2aadc06-84b2-4209-913d-c5c78fde951d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:46.735338 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f2aadc06-84b2-4209-913d-c5c78fde951d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3057271a87e04c639ceaf0657d0c6625', 'request_id': 'req-f2aadc06-84b2-4209-913d-c5c78fde951d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:46.756494 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f2aadc06-84b2-4209-913d-c5c78fde951d admin admin] The token cache is being invalidate because project 3057271a87e04c639ceaf0657d0c6625 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:46.758407 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f2aadc06-84b2-4209-913d-c5c78fde951d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:46.759084 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f2aadc06-84b2-4209-913d-c5c78fde951d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:46.760814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1319/2641] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:46 2022] DELETE /identity/v3/projects/3057271a87e04c639ceaf0657d0c6625 => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:46.765919 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2fef53-2c24-41ea-8f70-39f4c63728a0 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:46.766817 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2fef53-2c24-41ea-8f70-39f4c63728a0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:46.767369 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2fef53-2c24-41ea-8f70-39f4c63728a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:46.776362 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:46.788785 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1a2fef53-2c24-41ea-8f70-39f4c63728a0 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:46.790120 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1a2fef53-2c24-41ea-8f70-39f4c63728a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:46.864698 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1323/2642] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:46.866486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109d1795-9c62-4f53-9cee-137f76925872 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:46.866842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109d1795-9c62-4f53-9cee-137f76925872 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:46.867296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109d1795-9c62-4f53-9cee-137f76925872 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:46.897032 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-109d1795-9c62-4f53-9cee-137f76925872 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:46.898367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-109d1795-9c62-4f53-9cee-137f76925872 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:46.909618 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1320/2643] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:46.913576 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2fef53-2c24-41ea-8f70-39f4c63728a0 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:46.929620 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2fef53-2c24-41ea-8f70-39f4c63728a0 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:46.937128 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44feade-206e-46b9-864e-7ccccba26afc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:46.938020 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44feade-206e-46b9-864e-7ccccba26afc admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:46.939108 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44feade-206e-46b9-864e-7ccccba26afc admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:46.940181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44feade-206e-46b9-864e-7ccccba26afc admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:46.941288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44feade-206e-46b9-864e-7ccccba26afc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:56:46.941917 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44feade-206e-46b9-864e-7ccccba26afc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:46.944711 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44feade-206e-46b9-864e-7ccccba26afc admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:46.969952 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1324/2644] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:56:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:56:46.971161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109d1795-9c62-4f53-9cee-137f76925872 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:46.988026 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109d1795-9c62-4f53-9cee-137f76925872 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:46.995656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a1c823-13b8-4562-b7fa-3f390e67fe30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:46.996410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a1c823-13b8-4562-b7fa-3f390e67fe30 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:46.996861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a1c823-13b8-4562-b7fa-3f390e67fe30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:46.997202 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a1c823-13b8-4562-b7fa-3f390e67fe30 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:46.997780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a1c823-13b8-4562-b7fa-3f390e67fe30 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:46.999751 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a1c823-13b8-4562-b7fa-3f390e67fe30 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:47.166467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1321/2645] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:46 2022] POST /identity/v3/projects => generated 385 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:47.168650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44feade-206e-46b9-864e-7ccccba26afc admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:47.178480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44feade-206e-46b9-864e-7ccccba26afc admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:47.252791 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b615841c-89ed-419e-b5a7-dc52e5564481 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:47.253427 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b615841c-89ed-419e-b5a7-dc52e5564481 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:47.253799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b615841c-89ed-419e-b5a7-dc52e5564481 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:47.254073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b615841c-89ed-419e-b5a7-dc52e5564481 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:47.254781 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b615841c-89ed-419e-b5a7-dc52e5564481 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:47.256770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b615841c-89ed-419e-b5a7-dc52e5564481 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:47.385812 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1325/2646] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:47 2022] POST /identity/v3/users => generated 293 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:47.390766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a1c823-13b8-4562-b7fa-3f390e67fe30 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:47.401877 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a1c823-13b8-4562-b7fa-3f390e67fe30 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:47.407359 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4fe7c1-80be-431c-bfed-2a450889c168 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:47.407972 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4fe7c1-80be-431c-bfed-2a450889c168 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:47.408252 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4fe7c1-80be-431c-bfed-2a450889c168 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:47.408526 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4fe7c1-80be-431c-bfed-2a450889c168 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:47.409175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4fe7c1-80be-431c-bfed-2a450889c168 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:47.410768 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4fe7c1-80be-431c-bfed-2a450889c168 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:47.423093 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1322/2647] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:47 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:47.433209 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b615841c-89ed-419e-b5a7-dc52e5564481 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:47.439635 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b615841c-89ed-419e-b5a7-dc52e5564481 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:47.445863 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e08ffdf-3a93-461d-9fe3-660eb8349894 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:47.446514 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e08ffdf-3a93-461d-9fe3-660eb8349894 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:47.446792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e08ffdf-3a93-461d-9fe3-660eb8349894 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:47.447102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e08ffdf-3a93-461d-9fe3-660eb8349894 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:47.447754 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e08ffdf-3a93-461d-9fe3-660eb8349894 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:47.449846 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e08ffdf-3a93-461d-9fe3-660eb8349894 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:47.461604 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1326/2648] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:47 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:47.467280 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4fe7c1-80be-431c-bfed-2a450889c168 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:47.479739 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4fe7c1-80be-431c-bfed-2a450889c168 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:47.489404 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e9770a-dca3-4d64-9fbc-09c23fe82376 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:47.490600 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e9770a-dca3-4d64-9fbc-09c23fe82376 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:47.490884 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e9770a-dca3-4d64-9fbc-09c23fe82376 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:47.491190 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e9770a-dca3-4d64-9fbc-09c23fe82376 admin admin] PATH_INFO: `/v3/projects/e99bf31d116347c484f2eb4b3cdcc63e/users/676064caec324a0fa9dfbfa81780ef0f/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:47.507897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e9770a-dca3-4d64-9fbc-09c23fe82376 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e99bf31d116347c484f2eb4b3cdcc63e, user_id=676064caec324a0fa9dfbfa81780ef0f, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:47.510543 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e9770a-dca3-4d64-9fbc-09c23fe82376 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:47.665299 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1323/2649] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:47 2022] PUT /identity/v3/projects/e99bf31d116347c484f2eb4b3cdcc63e/users/676064caec324a0fa9dfbfa81780ef0f/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:47.677572 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d69859-ab88-400a-9263-0f371fbbb70b None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:47.677793 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d69859-ab88-400a-9263-0f371fbbb70b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:47.678070 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d69859-ab88-400a-9263-0f371fbbb70b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:47.714038 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-57d69859-ab88-400a-9263-0f371fbbb70b None None] MFA Rules not processed for user `676064caec324a0fa9dfbfa81780ef0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:47.714038 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-57d69859-ab88-400a-9263-0f371fbbb70b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:47.799086 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:47.802615 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1327/2650] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:47 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:47.807463 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e9770a-dca3-4d64-9fbc-09c23fe82376 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:47.883206 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e9770a-dca3-4d64-9fbc-09c23fe82376 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:47.890376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d267e4f-f168-49c0-977e-b467db893213 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:47.891053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d267e4f-f168-49c0-977e-b467db893213 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:47.891335 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d267e4f-f168-49c0-977e-b467db893213 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:47.891606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d267e4f-f168-49c0-977e-b467db893213 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:47.893112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d267e4f-f168-49c0-977e-b467db893213 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:47.895059 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d267e4f-f168-49c0-977e-b467db893213 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:48.077861 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1324/2651] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:47 2022] POST /identity/v3/projects => generated 387 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:48.081939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d69859-ab88-400a-9263-0f371fbbb70b None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:48.097915 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d69859-ab88-400a-9263-0f371fbbb70b None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:48.171180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9310c9-ae8f-4e1e-a296-2fc306b7fcfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:48.172345 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9310c9-ae8f-4e1e-a296-2fc306b7fcfd admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:48.172345 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9310c9-ae8f-4e1e-a296-2fc306b7fcfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:48.172700 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9310c9-ae8f-4e1e-a296-2fc306b7fcfd admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:48.173389 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9310c9-ae8f-4e1e-a296-2fc306b7fcfd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:48.175544 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9310c9-ae8f-4e1e-a296-2fc306b7fcfd admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:48.274129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1328/2652] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:48 2022] POST /identity/v3/users => generated 294 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:48.284645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d267e4f-f168-49c0-977e-b467db893213 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:48.298889 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d267e4f-f168-49c0-977e-b467db893213 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:48.304387 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d452541-bd13-438d-a530-4e00524479d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:48.305010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d452541-bd13-438d-a530-4e00524479d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:48.305477 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d452541-bd13-438d-a530-4e00524479d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:48.305764 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d452541-bd13-438d-a530-4e00524479d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:48.307534 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d452541-bd13-438d-a530-4e00524479d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:48.310354 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d452541-bd13-438d-a530-4e00524479d9 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:48.328711 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9310c9-ae8f-4e1e-a296-2fc306b7fcfd admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:48.331228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1325/2653] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:48 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:48.346033 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9310c9-ae8f-4e1e-a296-2fc306b7fcfd admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:48.352174 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc7bf6a-fa7d-4b74-8fca-7b32bddd8d30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:48.352797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc7bf6a-fa7d-4b74-8fca-7b32bddd8d30 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:48.353080 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc7bf6a-fa7d-4b74-8fca-7b32bddd8d30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:48.353543 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc7bf6a-fa7d-4b74-8fca-7b32bddd8d30 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:48.354197 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc7bf6a-fa7d-4b74-8fca-7b32bddd8d30 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:48.356649 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc7bf6a-fa7d-4b74-8fca-7b32bddd8d30 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:48.369262 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1329/2654] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:48 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:48.373540 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d452541-bd13-438d-a530-4e00524479d9 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:48.383891 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d452541-bd13-438d-a530-4e00524479d9 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:48.392234 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11c77ae-09c2-4c11-92a3-442d91d34368 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:48.396187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11c77ae-09c2-4c11-92a3-442d91d34368 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:48.396187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11c77ae-09c2-4c11-92a3-442d91d34368 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:48.396187 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11c77ae-09c2-4c11-92a3-442d91d34368 admin admin] PATH_INFO: `/v3/projects/9713667f808d4076a3760c9f794c41ef/users/400f8b81d5c14da094d11fa0a3359f9a/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:48.412571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11c77ae-09c2-4c11-92a3-442d91d34368 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9713667f808d4076a3760c9f794c41ef, user_id=400f8b81d5c14da094d11fa0a3359f9a, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:48.415221 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11c77ae-09c2-4c11-92a3-442d91d34368 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:48.513729 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1326/2655] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:48 2022] PUT /identity/v3/projects/9713667f808d4076a3760c9f794c41ef/users/400f8b81d5c14da094d11fa0a3359f9a/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:48.524435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f99cb2-4ef0-4721-ae7e-e66bceb80ea5 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:48.524816 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f99cb2-4ef0-4721-ae7e-e66bceb80ea5 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:48.525001 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f99cb2-4ef0-4721-ae7e-e66bceb80ea5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:48.570763 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-43f99cb2-4ef0-4721-ae7e-e66bceb80ea5 None None] MFA Rules not processed for user `400f8b81d5c14da094d11fa0a3359f9a`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:48.572335 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-43f99cb2-4ef0-4721-ae7e-e66bceb80ea5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:48.693062 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1330/2656] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:48 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:48.695784 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11c77ae-09c2-4c11-92a3-442d91d34368 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:48.722790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:48.792058 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11c77ae-09c2-4c11-92a3-442d91d34368 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:48.797618 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a657794b-5c33-40f4-8a41-7cdc56426184 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:48.798267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a657794b-5c33-40f4-8a41-7cdc56426184 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:48.798551 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a657794b-5c33-40f4-8a41-7cdc56426184 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:48.798828 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a657794b-5c33-40f4-8a41-7cdc56426184 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:48.799500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a657794b-5c33-40f4-8a41-7cdc56426184 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:48.801579 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a657794b-5c33-40f4-8a41-7cdc56426184 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:48.954719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1327/2657] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:48 2022] POST /identity/v3/projects => generated 387 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:48.956277 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f99cb2-4ef0-4721-ae7e-e66bceb80ea5 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:48.970692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f99cb2-4ef0-4721-ae7e-e66bceb80ea5 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:49.047676 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bada0f88-7dbb-4775-970c-3af3aa619285 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:49.048672 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bada0f88-7dbb-4775-970c-3af3aa619285 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:49.049293 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bada0f88-7dbb-4775-970c-3af3aa619285 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:49.049772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bada0f88-7dbb-4775-970c-3af3aa619285 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:49.050598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bada0f88-7dbb-4775-970c-3af3aa619285 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:49.052760 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bada0f88-7dbb-4775-970c-3af3aa619285 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:49.145460 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:49.158818 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1331/2658] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:48 2022] POST /identity/v3/users => generated 300 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:49.161332 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a657794b-5c33-40f4-8a41-7cdc56426184 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:49.172696 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a657794b-5c33-40f4-8a41-7cdc56426184 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:49.177353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b8e5ff-bd82-415a-ad1c-418b0ad9f589 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:49.178079 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b8e5ff-bd82-415a-ad1c-418b0ad9f589 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:49.178252 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b8e5ff-bd82-415a-ad1c-418b0ad9f589 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:49.178610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b8e5ff-bd82-415a-ad1c-418b0ad9f589 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:49.179291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b8e5ff-bd82-415a-ad1c-418b0ad9f589 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:49.180876 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b8e5ff-bd82-415a-ad1c-418b0ad9f589 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:49.202429 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1328/2659] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:49 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:49.202537 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bada0f88-7dbb-4775-970c-3af3aa619285 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:49.208116 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bada0f88-7dbb-4775-970c-3af3aa619285 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:49.212770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972bccbe-912d-4bb5-919e-b1ba3fff827e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:49.213426 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972bccbe-912d-4bb5-919e-b1ba3fff827e admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:49.213735 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972bccbe-912d-4bb5-919e-b1ba3fff827e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:49.214089 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972bccbe-912d-4bb5-919e-b1ba3fff827e admin admin] PATH_INFO: `/v3/projects/5d3edd26798f4889adf23d90fac9c96a/users/ff69eb2ed5c448a686e169e1a9c31869/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:49.229318 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972bccbe-912d-4bb5-919e-b1ba3fff827e admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d3edd26798f4889adf23d90fac9c96a, user_id=ff69eb2ed5c448a686e169e1a9c31869, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:49.232790 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972bccbe-912d-4bb5-919e-b1ba3fff827e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:49.339448 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1332/2660] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:49 2022] PUT /identity/v3/projects/5d3edd26798f4889adf23d90fac9c96a/users/ff69eb2ed5c448a686e169e1a9c31869/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:49.344148 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b8e5ff-bd82-415a-ad1c-418b0ad9f589 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:49.354384 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b8e5ff-bd82-415a-ad1c-418b0ad9f589 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:49.419106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70af52a-4a37-422c-9e25-aca395f34d48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:49.420520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70af52a-4a37-422c-9e25-aca395f34d48 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:49.420837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70af52a-4a37-422c-9e25-aca395f34d48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:49.421118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70af52a-4a37-422c-9e25-aca395f34d48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:49.421771 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70af52a-4a37-422c-9e25-aca395f34d48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:49.423499 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70af52a-4a37-422c-9e25-aca395f34d48 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:49.434750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1329/2661] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:49 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:49.444610 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972bccbe-912d-4bb5-919e-b1ba3fff827e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:49.453278 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972bccbe-912d-4bb5-919e-b1ba3fff827e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:49.458143 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89578bb3-676a-4fd8-847b-2e769c80fc22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:49.458823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89578bb3-676a-4fd8-847b-2e769c80fc22 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:49.459131 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89578bb3-676a-4fd8-847b-2e769c80fc22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:49.459407 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89578bb3-676a-4fd8-847b-2e769c80fc22 admin admin] PATH_INFO: `/v3/projects/5d3edd26798f4889adf23d90fac9c96a/users/ff69eb2ed5c448a686e169e1a9c31869/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:49.461360 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89578bb3-676a-4fd8-847b-2e769c80fc22 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d3edd26798f4889adf23d90fac9c96a, user_id=ff69eb2ed5c448a686e169e1a9c31869, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:49.464032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89578bb3-676a-4fd8-847b-2e769c80fc22 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:49.481309 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1333/2662] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:49 2022] PUT /identity/v3/projects/5d3edd26798f4889adf23d90fac9c96a/users/ff69eb2ed5c448a686e169e1a9c31869/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:49.490329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a71671-1b3c-428c-940c-3b48f4042f3c None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:49.490713 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a71671-1b3c-428c-940c-3b48f4042f3c None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:49.491014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a71671-1b3c-428c-940c-3b48f4042f3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:49.532248 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-a2a71671-1b3c-428c-940c-3b48f4042f3c None None] MFA Rules not processed for user `ff69eb2ed5c448a686e169e1a9c31869`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:49.533442 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a2a71671-1b3c-428c-940c-3b48f4042f3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:49.609353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1330/2663] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:49 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:49.620386 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89578bb3-676a-4fd8-847b-2e769c80fc22 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:49.623927 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-89578bb3-676a-4fd8-847b-2e769c80fc22 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:49.685858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89578bb3-676a-4fd8-847b-2e769c80fc22 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:49.690126 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1288ef8f-73f4-47fc-9c64-a85cdb3aceaf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:49.690749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1288ef8f-73f4-47fc-9c64-a85cdb3aceaf service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:49.691108 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1288ef8f-73f4-47fc-9c64-a85cdb3aceaf service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:49.691353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1288ef8f-73f4-47fc-9c64-a85cdb3aceaf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:49.701313 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1288ef8f-73f4-47fc-9c64-a85cdb3aceaf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:49.702756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1288ef8f-73f4-47fc-9c64-a85cdb3aceaf service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:49.712486 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1334/2664] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:49 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:50.610660 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a71671-1b3c-428c-940c-3b48f4042f3c None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:50.612112 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a2a71671-1b3c-428c-940c-3b48f4042f3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:50.622951 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:50.679831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a71671-1b3c-428c-940c-3b48f4042f3c None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:50.685137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9edf451-b153-4a8b-a574-253b8a69dc93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:50.685776 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9edf451-b153-4a8b-a574-253b8a69dc93 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:50.686058 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9edf451-b153-4a8b-a574-253b8a69dc93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:50.686335 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9edf451-b153-4a8b-a574-253b8a69dc93 admin admin] PATH_INFO: `/v3/users/411d8a02f2854e709beecf7a6a06dda0` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:50.687521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9edf451-b153-4a8b-a574-253b8a69dc93 admin admin] RBAC: Authorizing `identity:delete_user(user_id=411d8a02f2854e709beecf7a6a06dda0)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:50.692267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9edf451-b153-4a8b-a574-253b8a69dc93 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:50.896291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a9edf451-b153-4a8b-a574-253b8a69dc93 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '411d8a02f2854e709beecf7a6a06dda0'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:50.979065 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a9edf451-b153-4a8b-a574-253b8a69dc93 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '411d8a02f2854e709beecf7a6a06dda0'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:51.002695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a9edf451-b153-4a8b-a574-253b8a69dc93 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '411d8a02f2854e709beecf7a6a06dda0'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:51.003770 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a9edf451-b153-4a8b-a574-253b8a69dc93 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '411d8a02f2854e709beecf7a6a06dda0'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:51.009799 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1288ef8f-73f4-47fc-9c64-a85cdb3aceaf service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:51.012067 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1288ef8f-73f4-47fc-9c64-a85cdb3aceaf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:51.014698 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a9edf451-b153-4a8b-a574-253b8a69dc93 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '411d8a02f2854e709beecf7a6a06dda0'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:51.024787 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1288ef8f-73f4-47fc-9c64-a85cdb3aceaf service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:51.026006 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1331/2665] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:50 2022] DELETE /identity/v3/users/411d8a02f2854e709beecf7a6a06dda0 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:51.031837 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c223557-dac7-4f5e-bac6-4e4879ab878a None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:51.032602 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c223557-dac7-4f5e-bac6-4e4879ab878a None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:51.033166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c223557-dac7-4f5e-bac6-4e4879ab878a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:51.058156 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-9c223557-dac7-4f5e-bac6-4e4879ab878a None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:51.061565 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-9c223557-dac7-4f5e-bac6-4e4879ab878a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:51.092141 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b82ca57-4070-4beb-911f-72bf6e779cd2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:51.092688 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b82ca57-4070-4beb-911f-72bf6e779cd2 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:51.093035 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b82ca57-4070-4beb-911f-72bf6e779cd2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:51.093320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b82ca57-4070-4beb-911f-72bf6e779cd2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:51.095366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7b82ca57-4070-4beb-911f-72bf6e779cd2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:51.148486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1332/2666] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:51.163492 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c223557-dac7-4f5e-bac6-4e4879ab878a None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:51.166331 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-9c223557-dac7-4f5e-bac6-4e4879ab878a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:51.172345 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b82ca57-4070-4beb-911f-72bf6e779cd2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:51.173800 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b82ca57-4070-4beb-911f-72bf6e779cd2 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:51.186796 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1335/2667] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:51 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:56:51.214379 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:51.244723 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c223557-dac7-4f5e-bac6-4e4879ab878a None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:51.250506 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73dfc268-d15c-4187-9437-852365bab9a3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:51.251513 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73dfc268-d15c-4187-9437-852365bab9a3 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:51.251838 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73dfc268-d15c-4187-9437-852365bab9a3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:51.252282 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73dfc268-d15c-4187-9437-852365bab9a3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:51.264014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73dfc268-d15c-4187-9437-852365bab9a3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:51.265719 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73dfc268-d15c-4187-9437-852365bab9a3 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:51.279378 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1333/2668] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:56:51.914656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b82ca57-4070-4beb-911f-72bf6e779cd2 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:51.917186 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7b82ca57-4070-4beb-911f-72bf6e779cd2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:51.955030 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b82ca57-4070-4beb-911f-72bf6e779cd2 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:51.962699 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa6ad9f-c020-4577-8863-936d9972e5a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:51.963487 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa6ad9f-c020-4577-8863-936d9972e5a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:51.963843 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa6ad9f-c020-4577-8863-936d9972e5a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:51.964289 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa6ad9f-c020-4577-8863-936d9972e5a3 admin admin] PATH_INFO: `/v3/projects/404f951c8fa54b81a3692b541c47981c` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:51.966224 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa6ad9f-c020-4577-8863-936d9972e5a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=404f951c8fa54b81a3692b541c47981c)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:51.968273 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa6ad9f-c020-4577-8863-936d9972e5a3 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:51.985852 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:52.100986 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3fa6ad9f-c020-4577-8863-936d9972e5a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '404f951c8fa54b81a3692b541c47981c', 'request_id': 'req-3fa6ad9f-c020-4577-8863-936d9972e5a3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:52.107987 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3fa6ad9f-c020-4577-8863-936d9972e5a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '404f951c8fa54b81a3692b541c47981c', 'request_id': 'req-3fa6ad9f-c020-4577-8863-936d9972e5a3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:52.189525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3fa6ad9f-c020-4577-8863-936d9972e5a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '404f951c8fa54b81a3692b541c47981c', 'request_id': 'req-3fa6ad9f-c020-4577-8863-936d9972e5a3'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:52.210575 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3fa6ad9f-c020-4577-8863-936d9972e5a3 admin admin] The token cache is being invalidate because project 404f951c8fa54b81a3692b541c47981c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:52.210978 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3fa6ad9f-c020-4577-8863-936d9972e5a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:52.212004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3fa6ad9f-c020-4577-8863-936d9972e5a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:52.213523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1336/2669] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:51 2022] DELETE /identity/v3/projects/404f951c8fa54b81a3692b541c47981c => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:52.217854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73dfc268-d15c-4187-9437-852365bab9a3 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:52.234870 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73dfc268-d15c-4187-9437-852365bab9a3 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:52.292066 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a3f95b-c49a-423d-9c3b-c19dc40bf460 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:52.292708 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a3f95b-c49a-423d-9c3b-c19dc40bf460 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:52.294102 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a3f95b-c49a-423d-9c3b-c19dc40bf460 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:52.294600 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a3f95b-c49a-423d-9c3b-c19dc40bf460 admin admin] PATH_INFO: `/v3/users/9351d09333d7416d8e5b559e23217f80` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:52.295756 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a3f95b-c49a-423d-9c3b-c19dc40bf460 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9351d09333d7416d8e5b559e23217f80)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:52.297782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a3f95b-c49a-423d-9c3b-c19dc40bf460 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:52.517888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-39a3f95b-c49a-423d-9c3b-c19dc40bf460 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9351d09333d7416d8e5b559e23217f80'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:52.568621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-39a3f95b-c49a-423d-9c3b-c19dc40bf460 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9351d09333d7416d8e5b559e23217f80'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:52.590727 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-39a3f95b-c49a-423d-9c3b-c19dc40bf460 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9351d09333d7416d8e5b559e23217f80'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:52.591575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-39a3f95b-c49a-423d-9c3b-c19dc40bf460 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9351d09333d7416d8e5b559e23217f80'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:52.600639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-39a3f95b-c49a-423d-9c3b-c19dc40bf460 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9351d09333d7416d8e5b559e23217f80'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:52.610246 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1334/2670] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:52 2022] DELETE /identity/v3/users/9351d09333d7416d8e5b559e23217f80 => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:53.220964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa6ad9f-c020-4577-8863-936d9972e5a3 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:53.222584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3fa6ad9f-c020-4577-8863-936d9972e5a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:53.237091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa6ad9f-c020-4577-8863-936d9972e5a3 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:53.258702 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a3f95b-c49a-423d-9c3b-c19dc40bf460 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:53.261297 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-39a3f95b-c49a-423d-9c3b-c19dc40bf460 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:53.310580 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2582c591-fed9-4fc6-abc1-7a61bf39b6d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:53.311258 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2582c591-fed9-4fc6-abc1-7a61bf39b6d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:53.311757 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2582c591-fed9-4fc6-abc1-7a61bf39b6d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:53.312216 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2582c591-fed9-4fc6-abc1-7a61bf39b6d7 admin admin] PATH_INFO: `/v3/projects/217bb031aa714765866208831eba725e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:53.313997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2582c591-fed9-4fc6-abc1-7a61bf39b6d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=217bb031aa714765866208831eba725e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:53.316116 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2582c591-fed9-4fc6-abc1-7a61bf39b6d7 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:53.336205 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:53.350855 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a3f95b-c49a-423d-9c3b-c19dc40bf460 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:53.355573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5247bd-80b0-4faf-8548-13ea26774474 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:53.356204 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5247bd-80b0-4faf-8548-13ea26774474 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:53.356497 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5247bd-80b0-4faf-8548-13ea26774474 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:53.356766 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5247bd-80b0-4faf-8548-13ea26774474 admin admin] PATH_INFO: `/v3/users/269dbbd6d926496d8c5c704832de5763` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:53.357905 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5247bd-80b0-4faf-8548-13ea26774474 admin admin] RBAC: Authorizing `identity:delete_user(user_id=269dbbd6d926496d8c5c704832de5763)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:53.359863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5247bd-80b0-4faf-8548-13ea26774474 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:53.488136 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2582c591-fed9-4fc6-abc1-7a61bf39b6d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '217bb031aa714765866208831eba725e', 'request_id': 'req-2582c591-fed9-4fc6-abc1-7a61bf39b6d7'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:53.494480 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2582c591-fed9-4fc6-abc1-7a61bf39b6d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '217bb031aa714765866208831eba725e', 'request_id': 'req-2582c591-fed9-4fc6-abc1-7a61bf39b6d7'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:53.601769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2582c591-fed9-4fc6-abc1-7a61bf39b6d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '217bb031aa714765866208831eba725e', 'request_id': 'req-2582c591-fed9-4fc6-abc1-7a61bf39b6d7'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:53.613679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2582c591-fed9-4fc6-abc1-7a61bf39b6d7 admin admin] The token cache is being invalidate because project 217bb031aa714765866208831eba725e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:53.613999 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2582c591-fed9-4fc6-abc1-7a61bf39b6d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:53.614645 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2582c591-fed9-4fc6-abc1-7a61bf39b6d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:53.616235 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1337/2671] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:53 2022] DELETE /identity/v3/projects/217bb031aa714765866208831eba725e => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:53.620037 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2582c591-fed9-4fc6-abc1-7a61bf39b6d7 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:53.621614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-fe5247bd-80b0-4faf-8548-13ea26774474 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '269dbbd6d926496d8c5c704832de5763'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:53.621963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2582c591-fed9-4fc6-abc1-7a61bf39b6d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:53.655433 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:53.686463 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2582c591-fed9-4fc6-abc1-7a61bf39b6d7 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:53.691366 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b30c25b-9a38-46e2-87da-a139558e9339 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:53.691996 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b30c25b-9a38-46e2-87da-a139558e9339 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:53.692319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b30c25b-9a38-46e2-87da-a139558e9339 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:53.692589 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b30c25b-9a38-46e2-87da-a139558e9339 admin admin] PATH_INFO: `/v3/users/31a36f2b13f54341baf4ce3865d63f0d` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:53.693720 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b30c25b-9a38-46e2-87da-a139558e9339 admin admin] RBAC: Authorizing `identity:delete_user(user_id=31a36f2b13f54341baf4ce3865d63f0d)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:53.695728 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b30c25b-9a38-46e2-87da-a139558e9339 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:53.716449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-fe5247bd-80b0-4faf-8548-13ea26774474 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '269dbbd6d926496d8c5c704832de5763'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:53.729322 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-fe5247bd-80b0-4faf-8548-13ea26774474 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '269dbbd6d926496d8c5c704832de5763'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:53.729966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-fe5247bd-80b0-4faf-8548-13ea26774474 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '269dbbd6d926496d8c5c704832de5763'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:53.737351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-fe5247bd-80b0-4faf-8548-13ea26774474 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '269dbbd6d926496d8c5c704832de5763'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:53.743233 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1335/2672] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:53 2022] DELETE /identity/v3/users/269dbbd6d926496d8c5c704832de5763 => generated 0 bytes in 486 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:53.889221 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2b30c25b-9a38-46e2-87da-a139558e9339 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '31a36f2b13f54341baf4ce3865d63f0d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:53.901160 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2b30c25b-9a38-46e2-87da-a139558e9339 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '31a36f2b13f54341baf4ce3865d63f0d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:54.018564 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2b30c25b-9a38-46e2-87da-a139558e9339 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '31a36f2b13f54341baf4ce3865d63f0d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:54.019252 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2b30c25b-9a38-46e2-87da-a139558e9339 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '31a36f2b13f54341baf4ce3865d63f0d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:54.025200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5247bd-80b0-4faf-8548-13ea26774474 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:54.026793 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-fe5247bd-80b0-4faf-8548-13ea26774474 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:54.029682 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2b30c25b-9a38-46e2-87da-a139558e9339 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '31a36f2b13f54341baf4ce3865d63f0d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:54.037047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1338/2673] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:53 2022] DELETE /identity/v3/users/31a36f2b13f54341baf4ce3865d63f0d => generated 0 bytes in 417 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:56:54.110376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5247bd-80b0-4faf-8548-13ea26774474 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:54.170174 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f6cbc4-b399-4460-85df-50c3834312fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:54.170801 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f6cbc4-b399-4460-85df-50c3834312fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:54.171110 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f6cbc4-b399-4460-85df-50c3834312fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:54.171389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f6cbc4-b399-4460-85df-50c3834312fd admin admin] PATH_INFO: `/v3/projects/c705c2f92c1646dba9c11aa6000166bc` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:54.172665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f6cbc4-b399-4460-85df-50c3834312fd admin admin] RBAC: Authorizing `identity:delete_project(project_id=c705c2f92c1646dba9c11aa6000166bc)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:54.174559 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f6cbc4-b399-4460-85df-50c3834312fd admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:54.197203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:54.298760 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c6f6cbc4-b399-4460-85df-50c3834312fd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c705c2f92c1646dba9c11aa6000166bc', 'request_id': 'req-c6f6cbc4-b399-4460-85df-50c3834312fd'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:54.406686 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c6f6cbc4-b399-4460-85df-50c3834312fd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c705c2f92c1646dba9c11aa6000166bc', 'request_id': 'req-c6f6cbc4-b399-4460-85df-50c3834312fd'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:54.412613 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c6f6cbc4-b399-4460-85df-50c3834312fd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c705c2f92c1646dba9c11aa6000166bc', 'request_id': 'req-c6f6cbc4-b399-4460-85df-50c3834312fd'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:54.424574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c6f6cbc4-b399-4460-85df-50c3834312fd admin admin] The token cache is being invalidate because project c705c2f92c1646dba9c11aa6000166bc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:54.424862 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c6f6cbc4-b399-4460-85df-50c3834312fd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:54.425488 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c6f6cbc4-b399-4460-85df-50c3834312fd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:54.426985 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1336/2674] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:54 2022] DELETE /identity/v3/projects/c705c2f92c1646dba9c11aa6000166bc => generated 0 bytes in 402 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:54.431524 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b30c25b-9a38-46e2-87da-a139558e9339 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:54.434325 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2b30c25b-9a38-46e2-87da-a139558e9339 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:54.451773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b30c25b-9a38-46e2-87da-a139558e9339 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:54.462243 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f6cbc4-b399-4460-85df-50c3834312fd admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:54.463831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c6f6cbc4-b399-4460-85df-50c3834312fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:54.566014 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44b8346-3ac7-425d-aa38-e339b2c1a3d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:54.566650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44b8346-3ac7-425d-aa38-e339b2c1a3d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:54.567073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44b8346-3ac7-425d-aa38-e339b2c1a3d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:54.567446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44b8346-3ac7-425d-aa38-e339b2c1a3d8 admin admin] PATH_INFO: `/v3/users/d21a2934705941ceb91e2f33a4024be1` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:54.568694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44b8346-3ac7-425d-aa38-e339b2c1a3d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d21a2934705941ceb91e2f33a4024be1)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:54.570769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44b8346-3ac7-425d-aa38-e339b2c1a3d8 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:54.573016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f6cbc4-b399-4460-85df-50c3834312fd admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:54.579131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72dbe9fb-693d-4757-8633-385b5bb39e2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:54.579687 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72dbe9fb-693d-4757-8633-385b5bb39e2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:54.580368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72dbe9fb-693d-4757-8633-385b5bb39e2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:54.580368 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72dbe9fb-693d-4757-8633-385b5bb39e2c admin admin] PATH_INFO: `/v3/projects/08a4fc4962d144ffa5d50f8387431718` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:54.581555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72dbe9fb-693d-4757-8633-385b5bb39e2c admin admin] RBAC: Authorizing `identity:delete_project(project_id=08a4fc4962d144ffa5d50f8387431718)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:54.583612 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72dbe9fb-693d-4757-8633-385b5bb39e2c admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:54.803106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f44b8346-3ac7-425d-aa38-e339b2c1a3d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd21a2934705941ceb91e2f33a4024be1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:54.803465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-72dbe9fb-693d-4757-8633-385b5bb39e2c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '08a4fc4962d144ffa5d50f8387431718', 'request_id': 'req-72dbe9fb-693d-4757-8633-385b5bb39e2c'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:54.807680 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:54.819896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f44b8346-3ac7-425d-aa38-e339b2c1a3d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd21a2934705941ceb91e2f33a4024be1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:54.898723 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-72dbe9fb-693d-4757-8633-385b5bb39e2c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '08a4fc4962d144ffa5d50f8387431718', 'request_id': 'req-72dbe9fb-693d-4757-8633-385b5bb39e2c'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:54.905567 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-72dbe9fb-693d-4757-8633-385b5bb39e2c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '08a4fc4962d144ffa5d50f8387431718', 'request_id': 'req-72dbe9fb-693d-4757-8633-385b5bb39e2c'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:54.916846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-72dbe9fb-693d-4757-8633-385b5bb39e2c admin admin] The token cache is being invalidate because project 08a4fc4962d144ffa5d50f8387431718 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:54.917143 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-72dbe9fb-693d-4757-8633-385b5bb39e2c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:54.917769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-72dbe9fb-693d-4757-8633-385b5bb39e2c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:54.920264 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1337/2675] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:54 2022] DELETE /identity/v3/projects/08a4fc4962d144ffa5d50f8387431718 => generated 0 bytes in 458 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:54.924821 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2443959-643c-4037-9198-b704fd962ffc None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:54.925111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2443959-643c-4037-9198-b704fd962ffc None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:54.925393 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2443959-643c-4037-9198-b704fd962ffc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:54.945979 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-a2443959-643c-4037-9198-b704fd962ffc None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:54.947214 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a2443959-643c-4037-9198-b704fd962ffc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:54.982019 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:54.996299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f44b8346-3ac7-425d-aa38-e339b2c1a3d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd21a2934705941ceb91e2f33a4024be1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:54.997102 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f44b8346-3ac7-425d-aa38-e339b2c1a3d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd21a2934705941ceb91e2f33a4024be1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:55.004419 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f44b8346-3ac7-425d-aa38-e339b2c1a3d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd21a2934705941ceb91e2f33a4024be1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:55.008887 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1338/2676] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:56:55.011263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1339/2677] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:54 2022] DELETE /identity/v3/users/d21a2934705941ceb91e2f33a4024be1 => generated 0 bytes in 581 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:56:55.018518 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cde74b8-d8eb-4b39-8d90-23f859629d44 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:55.018799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cde74b8-d8eb-4b39-8d90-23f859629d44 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:55.019104 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cde74b8-d8eb-4b39-8d90-23f859629d44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:55.039309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-9cde74b8-d8eb-4b39-8d90-23f859629d44 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:55.040502 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-9cde74b8-d8eb-4b39-8d90-23f859629d44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:55.113349 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1339/2678] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:55.118244 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44b8346-3ac7-425d-aa38-e339b2c1a3d8 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:55.131990 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44b8346-3ac7-425d-aa38-e339b2c1a3d8 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:55.139515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26893d2-f0bf-4c7d-a3e0-7cae2d6678f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:55.140213 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26893d2-f0bf-4c7d-a3e0-7cae2d6678f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:55.140574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26893d2-f0bf-4c7d-a3e0-7cae2d6678f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:55.140854 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26893d2-f0bf-4c7d-a3e0-7cae2d6678f3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:55.141522 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26893d2-f0bf-4c7d-a3e0-7cae2d6678f3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:56:55.141815 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26893d2-f0bf-4c7d-a3e0-7cae2d6678f3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:55.143834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26893d2-f0bf-4c7d-a3e0-7cae2d6678f3 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:55.166617 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1340/2679] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:56:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:56:55.173593 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cde74b8-d8eb-4b39-8d90-23f859629d44 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:55.184523 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cde74b8-d8eb-4b39-8d90-23f859629d44 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:55.189754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf013acf-31d8-4451-ac66-6f504c6baa48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:55.190430 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf013acf-31d8-4451-ac66-6f504c6baa48 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:55.190718 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf013acf-31d8-4451-ac66-6f504c6baa48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:55.191036 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf013acf-31d8-4451-ac66-6f504c6baa48 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:55.191701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf013acf-31d8-4451-ac66-6f504c6baa48 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:55.193697 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf013acf-31d8-4451-ac66-6f504c6baa48 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:55.253778 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26893d2-f0bf-4c7d-a3e0-7cae2d6678f3 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:55.265635 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b26893d2-f0bf-4c7d-a3e0-7cae2d6678f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:55.271433 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26893d2-f0bf-4c7d-a3e0-7cae2d6678f3 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:55.276619 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a172d114-1dee-430a-aeac-872aacf0f571 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:55.278783 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a172d114-1dee-430a-aeac-872aacf0f571 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:55.279118 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a172d114-1dee-430a-aeac-872aacf0f571 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:55.279393 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a172d114-1dee-430a-aeac-872aacf0f571 admin admin] PATH_INFO: `/v3/projects/0ff078cd8282494eb7dd5ff34d7bddcb` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:55.280615 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a172d114-1dee-430a-aeac-872aacf0f571 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ff078cd8282494eb7dd5ff34d7bddcb)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:55.283777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a172d114-1dee-430a-aeac-872aacf0f571 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:55.293904 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1340/2680] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:55 2022] POST /identity/v3/projects => generated 385 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:56:55.299288 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf013acf-31d8-4451-ac66-6f504c6baa48 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:55.382557 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf013acf-31d8-4451-ac66-6f504c6baa48 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:55.387884 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0255c7a7-98c3-4d1d-8f98-04a22de26a62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:55.388485 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0255c7a7-98c3-4d1d-8f98-04a22de26a62 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:55.388755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0255c7a7-98c3-4d1d-8f98-04a22de26a62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:55.389030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0255c7a7-98c3-4d1d-8f98-04a22de26a62 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:55.389657 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0255c7a7-98c3-4d1d-8f98-04a22de26a62 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:55.391744 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0255c7a7-98c3-4d1d-8f98-04a22de26a62 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:55.409853 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a172d114-1dee-430a-aeac-872aacf0f571 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ff078cd8282494eb7dd5ff34d7bddcb', 'request_id': 'req-a172d114-1dee-430a-aeac-872aacf0f571'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:55.416740 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a172d114-1dee-430a-aeac-872aacf0f571 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ff078cd8282494eb7dd5ff34d7bddcb', 'request_id': 'req-a172d114-1dee-430a-aeac-872aacf0f571'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:55.505337 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a172d114-1dee-430a-aeac-872aacf0f571 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ff078cd8282494eb7dd5ff34d7bddcb', 'request_id': 'req-a172d114-1dee-430a-aeac-872aacf0f571'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:55.513382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1341/2681] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:55 2022] POST /identity/v3/users => generated 293 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:55.517869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0255c7a7-98c3-4d1d-8f98-04a22de26a62 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:55.518681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a172d114-1dee-430a-aeac-872aacf0f571 admin admin] The token cache is being invalidate because project 0ff078cd8282494eb7dd5ff34d7bddcb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:55.519069 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a172d114-1dee-430a-aeac-872aacf0f571 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:55.519804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a172d114-1dee-430a-aeac-872aacf0f571 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:55.521573 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1341/2682] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:55 2022] DELETE /identity/v3/projects/0ff078cd8282494eb7dd5ff34d7bddcb => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:56:55.537740 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0255c7a7-98c3-4d1d-8f98-04a22de26a62 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:55.615172 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7870fd-51ae-4eb8-879b-4024d3156823 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:55.615805 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7870fd-51ae-4eb8-879b-4024d3156823 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:55.616089 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7870fd-51ae-4eb8-879b-4024d3156823 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:55.616529 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7870fd-51ae-4eb8-879b-4024d3156823 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:55.617200 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7870fd-51ae-4eb8-879b-4024d3156823 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:55.619017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7870fd-51ae-4eb8-879b-4024d3156823 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:55.634217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1342/2683] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:55 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:55.635386 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a172d114-1dee-430a-aeac-872aacf0f571 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:55.637609 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a172d114-1dee-430a-aeac-872aacf0f571 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:55.651522 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a172d114-1dee-430a-aeac-872aacf0f571 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:55.656446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a25aa89-f170-4b51-9246-bc44b8080ab4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:55.657074 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a25aa89-f170-4b51-9246-bc44b8080ab4 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:55.657759 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a25aa89-f170-4b51-9246-bc44b8080ab4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:55.657759 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a25aa89-f170-4b51-9246-bc44b8080ab4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:55.658287 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a25aa89-f170-4b51-9246-bc44b8080ab4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:55.659956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a25aa89-f170-4b51-9246-bc44b8080ab4 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:55.671654 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1342/2684] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:55 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:56:55.676238 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7870fd-51ae-4eb8-879b-4024d3156823 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:55.687750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7870fd-51ae-4eb8-879b-4024d3156823 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:55.693114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d781fd-8fd9-4cc1-bfbf-9385d0dfec75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:55.693329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d781fd-8fd9-4cc1-bfbf-9385d0dfec75 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:55.693609 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d781fd-8fd9-4cc1-bfbf-9385d0dfec75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:55.693894 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d781fd-8fd9-4cc1-bfbf-9385d0dfec75 admin admin] PATH_INFO: `/v3/projects/dbfc2f845bf646cfa501228dd4ee4473/users/198dab566d0c46278600fb0475d7a0d8/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:55.716622 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d781fd-8fd9-4cc1-bfbf-9385d0dfec75 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dbfc2f845bf646cfa501228dd4ee4473, user_id=198dab566d0c46278600fb0475d7a0d8, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:55.719582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d781fd-8fd9-4cc1-bfbf-9385d0dfec75 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:55.836352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1343/2685] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:55 2022] PUT /identity/v3/projects/dbfc2f845bf646cfa501228dd4ee4473/users/198dab566d0c46278600fb0475d7a0d8/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:55.871184 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:55.903274 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488b23cf-94b9-4308-be48-91c72e6c28ef None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:55.914708 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488b23cf-94b9-4308-be48-91c72e6c28ef None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:55.914708 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488b23cf-94b9-4308-be48-91c72e6c28ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:55.932933 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-488b23cf-94b9-4308-be48-91c72e6c28ef None None] MFA Rules not processed for user `198dab566d0c46278600fb0475d7a0d8`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:55.934498 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-488b23cf-94b9-4308-be48-91c72e6c28ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:56.003675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1343/2686] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:55 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:56.008188 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d781fd-8fd9-4cc1-bfbf-9385d0dfec75 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:56.071495 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d781fd-8fd9-4cc1-bfbf-9385d0dfec75 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:56.076272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84341e5c-39f9-47e1-a609-60b3d64b73f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:56.076868 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84341e5c-39f9-47e1-a609-60b3d64b73f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:56.077232 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84341e5c-39f9-47e1-a609-60b3d64b73f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:56.077501 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84341e5c-39f9-47e1-a609-60b3d64b73f1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:56.078130 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84341e5c-39f9-47e1-a609-60b3d64b73f1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:56.080037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84341e5c-39f9-47e1-a609-60b3d64b73f1 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:56.238051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1344/2687] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:56 2022] POST /identity/v3/projects => generated 383 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:56.239536 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488b23cf-94b9-4308-be48-91c72e6c28ef None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:56.253226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488b23cf-94b9-4308-be48-91c72e6c28ef None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:56.272082 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:DELETE=10,keystone:SELECT=88,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:56.326084 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42ce93b-4b0f-4831-b56f-60c0118e5d45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:56.326774 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42ce93b-4b0f-4831-b56f-60c0118e5d45 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:56.327429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42ce93b-4b0f-4831-b56f-60c0118e5d45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:56.327920 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42ce93b-4b0f-4831-b56f-60c0118e5d45 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:56.330024 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42ce93b-4b0f-4831-b56f-60c0118e5d45 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:56.333233 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42ce93b-4b0f-4831-b56f-60c0118e5d45 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:56.476229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1344/2688] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:56 2022] POST /identity/v3/users => generated 292 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:56:56.480462 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84341e5c-39f9-47e1-a609-60b3d64b73f1 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:56.490831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84341e5c-39f9-47e1-a609-60b3d64b73f1 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:56.495337 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5261f44f-6f4f-4325-b054-d865564a10e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:56.495927 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5261f44f-6f4f-4325-b054-d865564a10e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:56.497015 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5261f44f-6f4f-4325-b054-d865564a10e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:56.497522 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5261f44f-6f4f-4325-b054-d865564a10e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:56.498159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5261f44f-6f4f-4325-b054-d865564a10e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:56.500365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5261f44f-6f4f-4325-b054-d865564a10e2 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:56.510813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1345/2689] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:56 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:56.515690 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42ce93b-4b0f-4831-b56f-60c0118e5d45 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:56.525525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42ce93b-4b0f-4831-b56f-60c0118e5d45 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:56.530176 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037b5eec-0ef2-49c1-866f-713b8b88a644 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:56.530764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037b5eec-0ef2-49c1-866f-713b8b88a644 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:56.531077 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037b5eec-0ef2-49c1-866f-713b8b88a644 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:56.531368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037b5eec-0ef2-49c1-866f-713b8b88a644 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:56.532040 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037b5eec-0ef2-49c1-866f-713b8b88a644 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:56.534218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037b5eec-0ef2-49c1-866f-713b8b88a644 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:56.551734 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1345/2690] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:56 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:56.551901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5261f44f-6f4f-4325-b054-d865564a10e2 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:56.551901 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-5261f44f-6f4f-4325-b054-d865564a10e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:56.552678 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037b5eec-0ef2-49c1-866f-713b8b88a644 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:56.563523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037b5eec-0ef2-49c1-866f-713b8b88a644 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:56.568537 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251eab2c-ac85-4b6a-8cf9-2c18cbd9a1a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:56.569176 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251eab2c-ac85-4b6a-8cf9-2c18cbd9a1a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:56.569454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251eab2c-ac85-4b6a-8cf9-2c18cbd9a1a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:56.569755 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251eab2c-ac85-4b6a-8cf9-2c18cbd9a1a6 admin admin] PATH_INFO: `/v3/projects/d2d9550c7d824227a5189f0fe585209e/users/f337fb99ff4c4e80a87127969e1021d0/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:56.586131 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251eab2c-ac85-4b6a-8cf9-2c18cbd9a1a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d2d9550c7d824227a5189f0fe585209e, user_id=f337fb99ff4c4e80a87127969e1021d0, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:56.589263 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251eab2c-ac85-4b6a-8cf9-2c18cbd9a1a6 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:56.609856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5261f44f-6f4f-4325-b054-d865564a10e2 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:56.613493 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786b667e-c511-4839-ad5d-98748fa76a72 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:56.614091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786b667e-c511-4839-ad5d-98748fa76a72 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:56.614366 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786b667e-c511-4839-ad5d-98748fa76a72 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:56.614634 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786b667e-c511-4839-ad5d-98748fa76a72 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:56.616364 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-786b667e-c511-4839-ad5d-98748fa76a72 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:56.654217 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1346/2691] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:56 2022] PUT /identity/v3/projects/d2d9550c7d824227a5189f0fe585209e/users/f337fb99ff4c4e80a87127969e1021d0/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:56.664481 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15ea97d-0e57-489e-bb67-939283b5c85d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:56.664769 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15ea97d-0e57-489e-bb67-939283b5c85d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:56.665048 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15ea97d-0e57-489e-bb67-939283b5c85d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:56.669792 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:56.683119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786b667e-c511-4839-ad5d-98748fa76a72 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:56.685380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786b667e-c511-4839-ad5d-98748fa76a72 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:56.694665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1346/2692] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:56 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:56.696744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e15ea97d-0e57-489e-bb67-939283b5c85d None None] MFA Rules not processed for user `f337fb99ff4c4e80a87127969e1021d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:56.698031 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e15ea97d-0e57-489e-bb67-939283b5c85d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:56.774088 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1347/2693] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:56 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:56:56.778927 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786b667e-c511-4839-ad5d-98748fa76a72 service cinder] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:56.789679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786b667e-c511-4839-ad5d-98748fa76a72 service cinder] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:56.853270 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba57b81-b0f5-4830-8d4f-4beb6fe1db73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:56.853887 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba57b81-b0f5-4830-8d4f-4beb6fe1db73 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:56.854159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba57b81-b0f5-4830-8d4f-4beb6fe1db73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:56.854446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba57b81-b0f5-4830-8d4f-4beb6fe1db73 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:56.855170 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba57b81-b0f5-4830-8d4f-4beb6fe1db73 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:56.857270 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba57b81-b0f5-4830-8d4f-4beb6fe1db73 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:56.997157 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1347/2694] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:56 2022] POST /identity/v3/projects => generated 385 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:57.001432 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15ea97d-0e57-489e-bb67-939283b5c85d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:57.012650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15ea97d-0e57-489e-bb67-939283b5c85d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:57.069323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a20491-c066-4111-ad7c-8f2d797687fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:57.069938 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a20491-c066-4111-ad7c-8f2d797687fc admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:57.070223 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a20491-c066-4111-ad7c-8f2d797687fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:57.070492 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a20491-c066-4111-ad7c-8f2d797687fc admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:57.071167 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a20491-c066-4111-ad7c-8f2d797687fc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:57.073113 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a20491-c066-4111-ad7c-8f2d797687fc admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:57.083114 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:57.156249 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1348/2695] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:57 2022] POST /identity/v3/users => generated 299 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:57.160283 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba57b81-b0f5-4830-8d4f-4beb6fe1db73 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:57.170437 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba57b81-b0f5-4830-8d4f-4beb6fe1db73 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:57.175472 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5801f3c-13d5-4c4a-89dc-dde6032e9acb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:57.176185 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5801f3c-13d5-4c4a-89dc-dde6032e9acb admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:57.176466 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5801f3c-13d5-4c4a-89dc-dde6032e9acb admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:57.176760 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5801f3c-13d5-4c4a-89dc-dde6032e9acb admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:57.177392 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5801f3c-13d5-4c4a-89dc-dde6032e9acb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:57.178990 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5801f3c-13d5-4c4a-89dc-dde6032e9acb admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:57.190460 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1348/2696] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:57.194616 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a20491-c066-4111-ad7c-8f2d797687fc admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:57.205598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a20491-c066-4111-ad7c-8f2d797687fc admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:57.210176 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cc5b6f-1897-4ad9-b1c1-be9fe988bcf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:57.210829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cc5b6f-1897-4ad9-b1c1-be9fe988bcf9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:57.211162 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cc5b6f-1897-4ad9-b1c1-be9fe988bcf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:57.211434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cc5b6f-1897-4ad9-b1c1-be9fe988bcf9 admin admin] PATH_INFO: `/v3/projects/9a8c35756a5c41cda69587a6f45fa91a/users/7bb9cca350f94c4ba5d4451db5d7bb95/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:57.225698 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cc5b6f-1897-4ad9-b1c1-be9fe988bcf9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a8c35756a5c41cda69587a6f45fa91a, user_id=7bb9cca350f94c4ba5d4451db5d7bb95, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:57.228384 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cc5b6f-1897-4ad9-b1c1-be9fe988bcf9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:57.291110 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5801f3c-13d5-4c4a-89dc-dde6032e9acb admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:57.292883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b5801f3c-13d5-4c4a-89dc-dde6032e9acb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:57.305577 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5801f3c-13d5-4c4a-89dc-dde6032e9acb admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:57.311188 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae15ce6-1426-4cfb-b71a-422ef51a5d50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:57.311817 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae15ce6-1426-4cfb-b71a-422ef51a5d50 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:57.312097 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae15ce6-1426-4cfb-b71a-422ef51a5d50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:57.312371 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae15ce6-1426-4cfb-b71a-422ef51a5d50 admin admin] PATH_INFO: `/v3/users/b970f9cb8fe444f3a25fac913f23f84d` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:57.313515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae15ce6-1426-4cfb-b71a-422ef51a5d50 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b970f9cb8fe444f3a25fac913f23f84d)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:57.315436 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae15ce6-1426-4cfb-b71a-422ef51a5d50 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:57.374585 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1349/2697] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:57 2022] PUT /identity/v3/projects/9a8c35756a5c41cda69587a6f45fa91a/users/7bb9cca350f94c4ba5d4451db5d7bb95/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:57.380001 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cc5b6f-1897-4ad9-b1c1-be9fe988bcf9 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:57.443004 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cc5b6f-1897-4ad9-b1c1-be9fe988bcf9 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:57.447891 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9e971a-8c34-40a7-851c-43b5fb604c46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:57.448500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9e971a-8c34-40a7-851c-43b5fb604c46 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:57.448787 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9e971a-8c34-40a7-851c-43b5fb604c46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:57.449065 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9e971a-8c34-40a7-851c-43b5fb604c46 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:57.449821 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9e971a-8c34-40a7-851c-43b5fb604c46 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:57.451448 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9e971a-8c34-40a7-851c-43b5fb604c46 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:57.461408 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1350/2698] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:57 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:56:57.465682 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9e971a-8c34-40a7-851c-43b5fb604c46 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:57.475246 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9e971a-8c34-40a7-851c-43b5fb604c46 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:57.479903 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde8a4e4-fed2-4299-b989-2991ff7d221e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:57.481155 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde8a4e4-fed2-4299-b989-2991ff7d221e admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:57.481494 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde8a4e4-fed2-4299-b989-2991ff7d221e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:57.481772 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde8a4e4-fed2-4299-b989-2991ff7d221e admin admin] PATH_INFO: `/v3/projects/9a8c35756a5c41cda69587a6f45fa91a/users/7bb9cca350f94c4ba5d4451db5d7bb95/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:57.483726 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde8a4e4-fed2-4299-b989-2991ff7d221e admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a8c35756a5c41cda69587a6f45fa91a, user_id=7bb9cca350f94c4ba5d4451db5d7bb95, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:57.486286 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde8a4e4-fed2-4299-b989-2991ff7d221e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:57.497090 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1351/2699] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:57 2022] PUT /identity/v3/projects/9a8c35756a5c41cda69587a6f45fa91a/users/7bb9cca350f94c4ba5d4451db5d7bb95/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:56:57.508562 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fa052d-0251-48f9-94bf-300cd9f0f2d9 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:57.508850 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fa052d-0251-48f9-94bf-300cd9f0f2d9 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:57.509131 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fa052d-0251-48f9-94bf-300cd9f0f2d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:57.536970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-53fa052d-0251-48f9-94bf-300cd9f0f2d9 None None] MFA Rules not processed for user `7bb9cca350f94c4ba5d4451db5d7bb95`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:57.538177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-53fa052d-0251-48f9-94bf-300cd9f0f2d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:57.609565 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1352/2700] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:57 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:56:57.620819 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53fa052d-0251-48f9-94bf-300cd9f0f2d9 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:57.622812 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-53fa052d-0251-48f9-94bf-300cd9f0f2d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:57.639154 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6ae15ce6-1426-4cfb-b71a-422ef51a5d50 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b970f9cb8fe444f3a25fac913f23f84d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:57.673411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:57.697434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53fa052d-0251-48f9-94bf-300cd9f0f2d9 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:57.702396 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff11665c-091b-4606-b98f-649717fa1f93 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:57.703116 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff11665c-091b-4606-b98f-649717fa1f93 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:57.703424 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff11665c-091b-4606-b98f-649717fa1f93 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:57.703733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff11665c-091b-4606-b98f-649717fa1f93 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:57.728864 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6ae15ce6-1426-4cfb-b71a-422ef51a5d50 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b970f9cb8fe444f3a25fac913f23f84d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:57.744659 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6ae15ce6-1426-4cfb-b71a-422ef51a5d50 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b970f9cb8fe444f3a25fac913f23f84d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:57.745365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6ae15ce6-1426-4cfb-b71a-422ef51a5d50 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b970f9cb8fe444f3a25fac913f23f84d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:57.753549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-6ae15ce6-1426-4cfb-b71a-422ef51a5d50 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b970f9cb8fe444f3a25fac913f23f84d'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:57.759540 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1349/2701] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:56:57 2022] DELETE /identity/v3/users/b970f9cb8fe444f3a25fac913f23f84d => generated 0 bytes in 469 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:56:57.770503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff11665c-091b-4606-b98f-649717fa1f93 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:57.771210 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d64e38-d77e-4ad8-971e-58fe3f280650 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:57.771510 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d64e38-d77e-4ad8-971e-58fe3f280650 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:57.772203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d64e38-d77e-4ad8-971e-58fe3f280650 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:57.772358 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff11665c-091b-4606-b98f-649717fa1f93 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:57.782521 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1353/2702] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:57 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:56:57.792136 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-a7d64e38-d77e-4ad8-971e-58fe3f280650 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:57.794536 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a7d64e38-d77e-4ad8-971e-58fe3f280650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:57.826013 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:57.861993 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1350/2703] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:57.871946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff11665c-091b-4606-b98f-649717fa1f93 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:57.873374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ff11665c-091b-4606-b98f-649717fa1f93 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:57.933917 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff11665c-091b-4606-b98f-649717fa1f93 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:57.985179 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec6ca7e-cfd9-4dfb-ba25-cc6e893bb1fe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:57.985791 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec6ca7e-cfd9-4dfb-ba25-cc6e893bb1fe service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:57.986083 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec6ca7e-cfd9-4dfb-ba25-cc6e893bb1fe service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:57.986344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec6ca7e-cfd9-4dfb-ba25-cc6e893bb1fe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:57.995348 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec6ca7e-cfd9-4dfb-ba25-cc6e893bb1fe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:57.996898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec6ca7e-cfd9-4dfb-ba25-cc6e893bb1fe service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:58.011133 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1354/2704] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:56:58.619788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d64e38-d77e-4ad8-971e-58fe3f280650 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:58.623919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a7d64e38-d77e-4ad8-971e-58fe3f280650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:58.653682 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d64e38-d77e-4ad8-971e-58fe3f280650 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:58.659529 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5042642f-def8-4fbe-ac97-c274fc257724 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:58.667964 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5042642f-def8-4fbe-ac97-c274fc257724 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:58.669329 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5042642f-def8-4fbe-ac97-c274fc257724 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:58.671732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5042642f-def8-4fbe-ac97-c274fc257724 admin admin] PATH_INFO: `/v3/projects/a9aebb23f7c644b1b36f4d19154490e0` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:58.674782 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5042642f-def8-4fbe-ac97-c274fc257724 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a9aebb23f7c644b1b36f4d19154490e0)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:58.678114 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5042642f-def8-4fbe-ac97-c274fc257724 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:58.803205 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5042642f-def8-4fbe-ac97-c274fc257724 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a9aebb23f7c644b1b36f4d19154490e0', 'request_id': 'req-5042642f-def8-4fbe-ac97-c274fc257724'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:58.864963 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5042642f-def8-4fbe-ac97-c274fc257724 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a9aebb23f7c644b1b36f4d19154490e0', 'request_id': 'req-5042642f-def8-4fbe-ac97-c274fc257724'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:58.870889 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5042642f-def8-4fbe-ac97-c274fc257724 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a9aebb23f7c644b1b36f4d19154490e0', 'request_id': 'req-5042642f-def8-4fbe-ac97-c274fc257724'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:58.883341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5042642f-def8-4fbe-ac97-c274fc257724 admin admin] The token cache is being invalidate because project a9aebb23f7c644b1b36f4d19154490e0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:56:58.883637 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5042642f-def8-4fbe-ac97-c274fc257724 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:58.884321 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5042642f-def8-4fbe-ac97-c274fc257724 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:56:58.885846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1351/2705] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:56:58 2022] DELETE /identity/v3/projects/a9aebb23f7c644b1b36f4d19154490e0 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:56:58.892500 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf6e299-c72c-45b9-881a-5cf0b006fbe1 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:58.893007 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf6e299-c72c-45b9-881a-5cf0b006fbe1 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:58.893489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf6e299-c72c-45b9-881a-5cf0b006fbe1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:58.915823 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-6cf6e299-c72c-45b9-881a-5cf0b006fbe1 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:58.916966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6cf6e299-c72c-45b9-881a-5cf0b006fbe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:58.961923 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:58.976805 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1355/2706] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:58.985453 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff54c90-10c1-4028-aff0-ba557683e0ca None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:58.985739 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff54c90-10c1-4028-aff0-ba557683e0ca None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:58.986010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff54c90-10c1-4028-aff0-ba557683e0ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:59.006545 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-3ff54c90-10c1-4028-aff0-ba557683e0ca None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:59.007780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3ff54c90-10c1-4028-aff0-ba557683e0ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:59.016762 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1352/2707] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:59.020994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf6e299-c72c-45b9-881a-5cf0b006fbe1 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:59.031132 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf6e299-c72c-45b9-881a-5cf0b006fbe1 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:59.035833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04d76a0-8e18-417f-a929-a9d9c6068cc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:59.036434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04d76a0-8e18-417f-a929-a9d9c6068cc5 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:59.036715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04d76a0-8e18-417f-a929-a9d9c6068cc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:59.036991 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04d76a0-8e18-417f-a929-a9d9c6068cc5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:59.037638 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04d76a0-8e18-417f-a929-a9d9c6068cc5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:56:59.037942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04d76a0-8e18-417f-a929-a9d9c6068cc5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:59.039566 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04d76a0-8e18-417f-a929-a9d9c6068cc5 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:59.050971 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1356/2708] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:56:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:56:59.060747 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff54c90-10c1-4028-aff0-ba557683e0ca None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:59.069984 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff54c90-10c1-4028-aff0-ba557683e0ca None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:59.075224 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39436d87-7fbf-41fc-9d6a-c6ca38d148a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:59.075821 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39436d87-7fbf-41fc-9d6a-c6ca38d148a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:59.076096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39436d87-7fbf-41fc-9d6a-c6ca38d148a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:59.076392 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39436d87-7fbf-41fc-9d6a-c6ca38d148a1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:59.077024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39436d87-7fbf-41fc-9d6a-c6ca38d148a1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:59.078961 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39436d87-7fbf-41fc-9d6a-c6ca38d148a1 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:59.180130 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1353/2709] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:56:59 2022] POST /identity/v3/projects => generated 383 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:59.184696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04d76a0-8e18-417f-a929-a9d9c6068cc5 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:59.194455 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04d76a0-8e18-417f-a929-a9d9c6068cc5 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:59.250641 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e470e1-6eb2-41f7-9bce-1345275eee7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:59.251278 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e470e1-6eb2-41f7-9bce-1345275eee7a admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:59.251556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e470e1-6eb2-41f7-9bce-1345275eee7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:59.251831 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e470e1-6eb2-41f7-9bce-1345275eee7a admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:59.252472 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e470e1-6eb2-41f7-9bce-1345275eee7a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:59.254346 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e470e1-6eb2-41f7-9bce-1345275eee7a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:59.356515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1357/2710] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:56:59 2022] POST /identity/v3/users => generated 292 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:56:59.360788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39436d87-7fbf-41fc-9d6a-c6ca38d148a1 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:59.371235 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39436d87-7fbf-41fc-9d6a-c6ca38d148a1 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:59.376094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2a5faf-f231-4119-be17-450abb6b11de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:59.376691 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2a5faf-f231-4119-be17-450abb6b11de admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:59.376969 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2a5faf-f231-4119-be17-450abb6b11de admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:59.377326 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2a5faf-f231-4119-be17-450abb6b11de admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:59.377969 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2a5faf-f231-4119-be17-450abb6b11de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:59.379604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2a5faf-f231-4119-be17-450abb6b11de admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:59.390142 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1354/2711] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:59 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:59.395914 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e470e1-6eb2-41f7-9bce-1345275eee7a admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:59.406623 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e470e1-6eb2-41f7-9bce-1345275eee7a admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:59.414881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18f3446-7d67-4086-ad0a-e295993f48ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:59.419309 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18f3446-7d67-4086-ad0a-e295993f48ef admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:59.419309 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18f3446-7d67-4086-ad0a-e295993f48ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:59.419309 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18f3446-7d67-4086-ad0a-e295993f48ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:59.419309 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18f3446-7d67-4086-ad0a-e295993f48ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:59.419309 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18f3446-7d67-4086-ad0a-e295993f48ef admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:59.430253 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1358/2712] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:56:59 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:56:59.434788 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2a5faf-f231-4119-be17-450abb6b11de admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:59.444927 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2a5faf-f231-4119-be17-450abb6b11de admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:59.449746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad2318b-e42e-47f3-b85e-b915372caed4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:59.453919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad2318b-e42e-47f3-b85e-b915372caed4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:59.453919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad2318b-e42e-47f3-b85e-b915372caed4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:59.453919 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad2318b-e42e-47f3-b85e-b915372caed4 admin admin] PATH_INFO: `/v3/projects/5f35241168b542c6a61aeb5065ca80a2/users/c659ce1eab064353989f3ffef55705b1/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:59.466380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad2318b-e42e-47f3-b85e-b915372caed4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f35241168b542c6a61aeb5065ca80a2, user_id=c659ce1eab064353989f3ffef55705b1, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:59.469156 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad2318b-e42e-47f3-b85e-b915372caed4 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:59.533745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1355/2713] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:56:59 2022] PUT /identity/v3/projects/5f35241168b542c6a61aeb5065ca80a2/users/c659ce1eab064353989f3ffef55705b1/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:56:59.545715 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bff402a-12d2-485f-bcf3-6f2f951de159 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:59.546001 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bff402a-12d2-485f-bcf3-6f2f951de159 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:59.546274 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bff402a-12d2-485f-bcf3-6f2f951de159 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:59.574039 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9bff402a-12d2-485f-bcf3-6f2f951de159 None None] MFA Rules not processed for user `c659ce1eab064353989f3ffef55705b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:56:59.575358 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9bff402a-12d2-485f-bcf3-6f2f951de159 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:59.638586 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1359/2714] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:56:59 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:56:59.649262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad2318b-e42e-47f3-b85e-b915372caed4 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:56:59.651012 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-fad2318b-e42e-47f3-b85e-b915372caed4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:56:59.708508 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad2318b-e42e-47f3-b85e-b915372caed4 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:56:59.712307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c462188-2e1e-42f3-8ac4-f322251f54b5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:56:59.712899 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c462188-2e1e-42f3-8ac4-f322251f54b5 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:56:59.713180 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c462188-2e1e-42f3-8ac4-f322251f54b5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:56:59.713453 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c462188-2e1e-42f3-8ac4-f322251f54b5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:56:59.720262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:56:59.722555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c462188-2e1e-42f3-8ac4-f322251f54b5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:56:59.724024 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c462188-2e1e-42f3-8ac4-f322251f54b5 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:56:59.733873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1356/2715] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:56:59 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:57:00.492586 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bff402a-12d2-485f-bcf3-6f2f951de159 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:00.503275 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bff402a-12d2-485f-bcf3-6f2f951de159 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:00.507256 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0cdd612-fa0c-407c-b3ce-8d23910bdc9a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:00.507916 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0cdd612-fa0c-407c-b3ce-8d23910bdc9a service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:00.508164 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0cdd612-fa0c-407c-b3ce-8d23910bdc9a service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:00.508434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0cdd612-fa0c-407c-b3ce-8d23910bdc9a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:00.511648 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f0cdd612-fa0c-407c-b3ce-8d23910bdc9a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:00.556606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:00.573729 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0cdd612-fa0c-407c-b3ce-8d23910bdc9a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:00.576531 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0cdd612-fa0c-407c-b3ce-8d23910bdc9a service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:00.584254 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1360/2716] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:00 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:57:01.305854 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c462188-2e1e-42f3-8ac4-f322251f54b5 service cinder] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:01.307912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1c462188-2e1e-42f3-8ac4-f322251f54b5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:01.375640 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c462188-2e1e-42f3-8ac4-f322251f54b5 service cinder] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:01.380290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca85003-63ca-4863-879f-60a830792c05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:01.380940 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca85003-63ca-4863-879f-60a830792c05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:01.381219 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca85003-63ca-4863-879f-60a830792c05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:01.381492 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca85003-63ca-4863-879f-60a830792c05 admin admin] PATH_INFO: `/v3/users/676064caec324a0fa9dfbfa81780ef0f` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:01.382613 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca85003-63ca-4863-879f-60a830792c05 admin admin] RBAC: Authorizing `identity:delete_user(user_id=676064caec324a0fa9dfbfa81780ef0f)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:01.384575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca85003-63ca-4863-879f-60a830792c05 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:01.583281 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-9ca85003-63ca-4863-879f-60a830792c05 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '676064caec324a0fa9dfbfa81780ef0f'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:01.658786 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-9ca85003-63ca-4863-879f-60a830792c05 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '676064caec324a0fa9dfbfa81780ef0f'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:01.673043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-9ca85003-63ca-4863-879f-60a830792c05 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '676064caec324a0fa9dfbfa81780ef0f'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:01.673715 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-9ca85003-63ca-4863-879f-60a830792c05 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '676064caec324a0fa9dfbfa81780ef0f'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:01.680592 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-9ca85003-63ca-4863-879f-60a830792c05 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '676064caec324a0fa9dfbfa81780ef0f'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:01.687043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1357/2717] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:57:01 2022] DELETE /identity/v3/users/676064caec324a0fa9dfbfa81780ef0f => generated 0 bytes in 382 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:01.691680 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5cd3dc-050d-4ff1-bfe6-967bf2c59a54 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:01.692191 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5cd3dc-050d-4ff1-bfe6-967bf2c59a54 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:01.692299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5cd3dc-050d-4ff1-bfe6-967bf2c59a54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:01.712909 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ac5cd3dc-050d-4ff1-bfe6-967bf2c59a54 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:01.715215 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ac5cd3dc-050d-4ff1-bfe6-967bf2c59a54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:01.774234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1361/2718] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:01.785028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca85003-63ca-4863-879f-60a830792c05 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:01.786403 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-9ca85003-63ca-4863-879f-60a830792c05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:01.844675 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca85003-63ca-4863-879f-60a830792c05 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:01.848380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79520129-a3bf-4fb3-a35c-90171c9cc064 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:01.848988 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79520129-a3bf-4fb3-a35c-90171c9cc064 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:01.849266 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79520129-a3bf-4fb3-a35c-90171c9cc064 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:01.849540 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79520129-a3bf-4fb3-a35c-90171c9cc064 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:01.859347 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79520129-a3bf-4fb3-a35c-90171c9cc064 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:01.864280 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79520129-a3bf-4fb3-a35c-90171c9cc064 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:01.870091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1358/2719] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:57:02.163524 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5cd3dc-050d-4ff1-bfe6-967bf2c59a54 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:02.165189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ac5cd3dc-050d-4ff1-bfe6-967bf2c59a54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:02.224992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5cd3dc-050d-4ff1-bfe6-967bf2c59a54 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:02.228549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafd62d0-e771-4e9d-aea3-73c465b48ad3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:02.229148 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafd62d0-e771-4e9d-aea3-73c465b48ad3 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:02.229422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafd62d0-e771-4e9d-aea3-73c465b48ad3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:02.229692 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafd62d0-e771-4e9d-aea3-73c465b48ad3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:02.231396 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fafd62d0-e771-4e9d-aea3-73c465b48ad3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:02.294815 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafd62d0-e771-4e9d-aea3-73c465b48ad3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:02.296359 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafd62d0-e771-4e9d-aea3-73c465b48ad3 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:02.299724 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79520129-a3bf-4fb3-a35c-90171c9cc064 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:02.301203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-79520129-a3bf-4fb3-a35c-90171c9cc064 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:02.307099 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1362/2720] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:02 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:57:02.327923 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79520129-a3bf-4fb3-a35c-90171c9cc064 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:02.333155 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adad88c2-3411-4c4a-bdd0-00e744a50969 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:02.333787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adad88c2-3411-4c4a-bdd0-00e744a50969 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:02.334071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adad88c2-3411-4c4a-bdd0-00e744a50969 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:02.334428 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adad88c2-3411-4c4a-bdd0-00e744a50969 admin admin] PATH_INFO: `/v3/projects/e99bf31d116347c484f2eb4b3cdcc63e` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:02.335669 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adad88c2-3411-4c4a-bdd0-00e744a50969 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e99bf31d116347c484f2eb4b3cdcc63e)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:02.337432 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adad88c2-3411-4c4a-bdd0-00e744a50969 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:02.351207 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:02.453789 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-adad88c2-3411-4c4a-bdd0-00e744a50969 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e99bf31d116347c484f2eb4b3cdcc63e', 'request_id': 'req-adad88c2-3411-4c4a-bdd0-00e744a50969'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:02.526010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-adad88c2-3411-4c4a-bdd0-00e744a50969 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e99bf31d116347c484f2eb4b3cdcc63e', 'request_id': 'req-adad88c2-3411-4c4a-bdd0-00e744a50969'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:02.532387 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-adad88c2-3411-4c4a-bdd0-00e744a50969 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e99bf31d116347c484f2eb4b3cdcc63e', 'request_id': 'req-adad88c2-3411-4c4a-bdd0-00e744a50969'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:02.544306 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-adad88c2-3411-4c4a-bdd0-00e744a50969 admin admin] The token cache is being invalidate because project e99bf31d116347c484f2eb4b3cdcc63e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:57:02.544888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-adad88c2-3411-4c4a-bdd0-00e744a50969 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:02.545947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-adad88c2-3411-4c4a-bdd0-00e744a50969 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:02.547499 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1359/2721] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:57:02 2022] DELETE /identity/v3/projects/e99bf31d116347c484f2eb4b3cdcc63e => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:57:02.554633 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafd62d0-e771-4e9d-aea3-73c465b48ad3 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:02.565753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafd62d0-e771-4e9d-aea3-73c465b48ad3 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:02.602669 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:02.628881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0863fb15-6a15-4c23-835a-eda479d58192 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:02.629740 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0863fb15-6a15-4c23-835a-eda479d58192 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:02.630026 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0863fb15-6a15-4c23-835a-eda479d58192 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:02.630303 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0863fb15-6a15-4c23-835a-eda479d58192 admin admin] PATH_INFO: `/v3/users/400f8b81d5c14da094d11fa0a3359f9a` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:02.631486 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0863fb15-6a15-4c23-835a-eda479d58192 admin admin] RBAC: Authorizing `identity:delete_user(user_id=400f8b81d5c14da094d11fa0a3359f9a)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:02.633382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0863fb15-6a15-4c23-835a-eda479d58192 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:02.837987 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0863fb15-6a15-4c23-835a-eda479d58192 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '400f8b81d5c14da094d11fa0a3359f9a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:02.850069 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0863fb15-6a15-4c23-835a-eda479d58192 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '400f8b81d5c14da094d11fa0a3359f9a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:02.944770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0863fb15-6a15-4c23-835a-eda479d58192 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '400f8b81d5c14da094d11fa0a3359f9a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:02.945391 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0863fb15-6a15-4c23-835a-eda479d58192 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '400f8b81d5c14da094d11fa0a3359f9a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:02.952900 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0863fb15-6a15-4c23-835a-eda479d58192 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '400f8b81d5c14da094d11fa0a3359f9a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:02.959023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1363/2722] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:57:02 2022] DELETE /identity/v3/users/400f8b81d5c14da094d11fa0a3359f9a => generated 0 bytes in 405 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:03.428137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adad88c2-3411-4c4a-bdd0-00e744a50969 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:03.429676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-adad88c2-3411-4c4a-bdd0-00e744a50969 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:03.447672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adad88c2-3411-4c4a-bdd0-00e744a50969 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:03.505964 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86022f9a-a33c-4b0f-82a8-2ee4cf62fe27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:03.506589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86022f9a-a33c-4b0f-82a8-2ee4cf62fe27 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:03.506866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86022f9a-a33c-4b0f-82a8-2ee4cf62fe27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:03.507161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86022f9a-a33c-4b0f-82a8-2ee4cf62fe27 admin admin] PATH_INFO: `/v3/projects/9713667f808d4076a3760c9f794c41ef` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:03.508254 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86022f9a-a33c-4b0f-82a8-2ee4cf62fe27 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9713667f808d4076a3760c9f794c41ef)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:03.510139 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86022f9a-a33c-4b0f-82a8-2ee4cf62fe27 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:03.619728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-86022f9a-a33c-4b0f-82a8-2ee4cf62fe27 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9713667f808d4076a3760c9f794c41ef', 'request_id': 'req-86022f9a-a33c-4b0f-82a8-2ee4cf62fe27'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:03.710346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-86022f9a-a33c-4b0f-82a8-2ee4cf62fe27 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9713667f808d4076a3760c9f794c41ef', 'request_id': 'req-86022f9a-a33c-4b0f-82a8-2ee4cf62fe27'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:03.717376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-86022f9a-a33c-4b0f-82a8-2ee4cf62fe27 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9713667f808d4076a3760c9f794c41ef', 'request_id': 'req-86022f9a-a33c-4b0f-82a8-2ee4cf62fe27'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:03.728696 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-86022f9a-a33c-4b0f-82a8-2ee4cf62fe27 admin admin] The token cache is being invalidate because project 9713667f808d4076a3760c9f794c41ef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:57:03.728986 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-86022f9a-a33c-4b0f-82a8-2ee4cf62fe27 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:03.729622 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-86022f9a-a33c-4b0f-82a8-2ee4cf62fe27 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:03.731133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1360/2723] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:57:03 2022] DELETE /identity/v3/projects/9713667f808d4076a3760c9f794c41ef => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:03.735392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0863fb15-6a15-4c23-835a-eda479d58192 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:03.736829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0863fb15-6a15-4c23-835a-eda479d58192 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:03.750086 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0863fb15-6a15-4c23-835a-eda479d58192 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:03.813140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d841e7-b3cb-42b5-87c5-cb5d408d562b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:03.813768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d841e7-b3cb-42b5-87c5-cb5d408d562b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:03.814045 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d841e7-b3cb-42b5-87c5-cb5d408d562b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:03.814317 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d841e7-b3cb-42b5-87c5-cb5d408d562b admin admin] PATH_INFO: `/v3/users/ff69eb2ed5c448a686e169e1a9c31869` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:03.815549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d841e7-b3cb-42b5-87c5-cb5d408d562b admin admin] RBAC: Authorizing `identity:delete_user(user_id=ff69eb2ed5c448a686e169e1a9c31869)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:03.817445 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d841e7-b3cb-42b5-87c5-cb5d408d562b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:03.947140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d1d841e7-b3cb-42b5-87c5-cb5d408d562b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ff69eb2ed5c448a686e169e1a9c31869'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:03.953133 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=81,keystone:DELETE=18,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:03.959918 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d1d841e7-b3cb-42b5-87c5-cb5d408d562b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ff69eb2ed5c448a686e169e1a9c31869'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:04.024441 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d1d841e7-b3cb-42b5-87c5-cb5d408d562b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ff69eb2ed5c448a686e169e1a9c31869'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:04.024984 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d1d841e7-b3cb-42b5-87c5-cb5d408d562b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ff69eb2ed5c448a686e169e1a9c31869'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:04.032219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d1d841e7-b3cb-42b5-87c5-cb5d408d562b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ff69eb2ed5c448a686e169e1a9c31869'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:04.038992 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1364/2724] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:57:03 2022] DELETE /identity/v3/users/ff69eb2ed5c448a686e169e1a9c31869 => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:04.445342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86022f9a-a33c-4b0f-82a8-2ee4cf62fe27 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:04.446859 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-86022f9a-a33c-4b0f-82a8-2ee4cf62fe27 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:04.460563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86022f9a-a33c-4b0f-82a8-2ee4cf62fe27 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:04.492835 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:04.521107 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48cd2ed-e194-491e-b865-4358f0d7dbde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:04.521758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48cd2ed-e194-491e-b865-4358f0d7dbde admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:04.522232 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48cd2ed-e194-491e-b865-4358f0d7dbde admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:04.522612 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48cd2ed-e194-491e-b865-4358f0d7dbde admin admin] PATH_INFO: `/v3/projects/5d3edd26798f4889adf23d90fac9c96a` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:04.524042 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48cd2ed-e194-491e-b865-4358f0d7dbde admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d3edd26798f4889adf23d90fac9c96a)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:04.526096 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48cd2ed-e194-491e-b865-4358f0d7dbde admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:04.641173 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-e48cd2ed-e194-491e-b865-4358f0d7dbde admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d3edd26798f4889adf23d90fac9c96a', 'request_id': 'req-e48cd2ed-e194-491e-b865-4358f0d7dbde'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:04.704656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-e48cd2ed-e194-491e-b865-4358f0d7dbde admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d3edd26798f4889adf23d90fac9c96a', 'request_id': 'req-e48cd2ed-e194-491e-b865-4358f0d7dbde'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:04.710800 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-e48cd2ed-e194-491e-b865-4358f0d7dbde admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d3edd26798f4889adf23d90fac9c96a', 'request_id': 'req-e48cd2ed-e194-491e-b865-4358f0d7dbde'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:04.726585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-e48cd2ed-e194-491e-b865-4358f0d7dbde admin admin] The token cache is being invalidate because project 5d3edd26798f4889adf23d90fac9c96a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:57:04.726873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-e48cd2ed-e194-491e-b865-4358f0d7dbde admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:04.727562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-e48cd2ed-e194-491e-b865-4358f0d7dbde admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:04.729057 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1361/2725] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:57:04 2022] DELETE /identity/v3/projects/5d3edd26798f4889adf23d90fac9c96a => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:04.735911 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac03e4b-a898-40d5-86c7-e9abf5400856 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:04.737960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac03e4b-a898-40d5-86c7-e9abf5400856 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:04.737960 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac03e4b-a898-40d5-86c7-e9abf5400856 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:04.755957 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7ac03e4b-a898-40d5-86c7-e9abf5400856 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:04.757155 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7ac03e4b-a898-40d5-86c7-e9abf5400856 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:04.814188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1365/2726] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:04.823974 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc18713-c4de-41cb-9723-8caa274404d2 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:04.824257 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc18713-c4de-41cb-9723-8caa274404d2 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:04.824530 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc18713-c4de-41cb-9723-8caa274404d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:04.844568 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-cdc18713-c4de-41cb-9723-8caa274404d2 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:04.845747 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-cdc18713-c4de-41cb-9723-8caa274404d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:04.853479 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1362/2727] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:04.858289 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac03e4b-a898-40d5-86c7-e9abf5400856 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:04.867869 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac03e4b-a898-40d5-86c7-e9abf5400856 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:04.872598 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64ed212-1e0a-422e-b0b0-2763838d277b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:04.873195 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64ed212-1e0a-422e-b0b0-2763838d277b admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:04.873476 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64ed212-1e0a-422e-b0b0-2763838d277b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:04.873753 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64ed212-1e0a-422e-b0b0-2763838d277b admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:04.874398 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64ed212-1e0a-422e-b0b0-2763838d277b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:57:04.874733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64ed212-1e0a-422e-b0b0-2763838d277b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:04.876360 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64ed212-1e0a-422e-b0b0-2763838d277b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:04.888343 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1366/2728] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:57:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:57:04.892650 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc18713-c4de-41cb-9723-8caa274404d2 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:04.901695 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc18713-c4de-41cb-9723-8caa274404d2 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:04.907157 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef63db79-baea-444b-9b69-45ad42ea4703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:04.907780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef63db79-baea-444b-9b69-45ad42ea4703 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:04.908060 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef63db79-baea-444b-9b69-45ad42ea4703 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:04.908363 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef63db79-baea-444b-9b69-45ad42ea4703 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:04.908979 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef63db79-baea-444b-9b69-45ad42ea4703 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:04.910859 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef63db79-baea-444b-9b69-45ad42ea4703 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:05.003212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1363/2729] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:57:04 2022] POST /identity/v3/projects => generated 409 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:57:05.007343 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64ed212-1e0a-422e-b0b0-2763838d277b admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:05.018224 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64ed212-1e0a-422e-b0b0-2763838d277b admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:05.076319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913a648b-0b76-496e-947a-2c852a3b9f46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:05.076935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913a648b-0b76-496e-947a-2c852a3b9f46 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:05.077221 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913a648b-0b76-496e-947a-2c852a3b9f46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:05.077496 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913a648b-0b76-496e-947a-2c852a3b9f46 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:05.078135 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913a648b-0b76-496e-947a-2c852a3b9f46 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:05.080087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913a648b-0b76-496e-947a-2c852a3b9f46 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:05.161959 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1367/2730] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:57:05 2022] POST /identity/v3/users => generated 305 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:57:05.178075 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef63db79-baea-444b-9b69-45ad42ea4703 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:05.188928 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef63db79-baea-444b-9b69-45ad42ea4703 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:05.193872 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a81a3e-f1c5-41e8-b640-99033437658c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:05.194464 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a81a3e-f1c5-41e8-b640-99033437658c admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:05.194743 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a81a3e-f1c5-41e8-b640-99033437658c admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:05.195042 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a81a3e-f1c5-41e8-b640-99033437658c admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:05.195674 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a81a3e-f1c5-41e8-b640-99033437658c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:05.197241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a81a3e-f1c5-41e8-b640-99033437658c admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:05.207880 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1364/2731] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:05 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:57:05.212088 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913a648b-0b76-496e-947a-2c852a3b9f46 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:05.223459 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913a648b-0b76-496e-947a-2c852a3b9f46 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:05.228138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1723123c-35fa-4594-bb6c-b6f899c94008 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:05.229073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1723123c-35fa-4594-bb6c-b6f899c94008 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:05.229073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1723123c-35fa-4594-bb6c-b6f899c94008 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:05.229288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1723123c-35fa-4594-bb6c-b6f899c94008 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:05.229956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1723123c-35fa-4594-bb6c-b6f899c94008 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:05.231858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1723123c-35fa-4594-bb6c-b6f899c94008 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:05.242806 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1368/2732] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:05 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:57:05.248256 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a81a3e-f1c5-41e8-b640-99033437658c admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:05.258435 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a81a3e-f1c5-41e8-b640-99033437658c admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:05.263595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a140fd-efe0-4e32-98d9-d00991e09f34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:05.264267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a140fd-efe0-4e32-98d9-d00991e09f34 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:05.264547 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a140fd-efe0-4e32-98d9-d00991e09f34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:05.264825 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a140fd-efe0-4e32-98d9-d00991e09f34 admin admin] PATH_INFO: `/v3/projects/df64a64f6a9a4f78b77c8b0e3b982a6b/users/5dcbee68f48f4b348954b6edc6d21b8f/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:05.279439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a140fd-efe0-4e32-98d9-d00991e09f34 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df64a64f6a9a4f78b77c8b0e3b982a6b, user_id=5dcbee68f48f4b348954b6edc6d21b8f, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:05.282150 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a140fd-efe0-4e32-98d9-d00991e09f34 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:05.357262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1365/2733] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:57:05 2022] PUT /identity/v3/projects/df64a64f6a9a4f78b77c8b0e3b982a6b/users/5dcbee68f48f4b348954b6edc6d21b8f/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:05.368747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bd1753-55ef-4b1a-81de-37af7cfde06f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:05.369051 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bd1753-55ef-4b1a-81de-37af7cfde06f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:05.369326 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bd1753-55ef-4b1a-81de-37af7cfde06f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:05.397824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-22bd1753-55ef-4b1a-81de-37af7cfde06f None None] MFA Rules not processed for user `5dcbee68f48f4b348954b6edc6d21b8f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:05.399048 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-22bd1753-55ef-4b1a-81de-37af7cfde06f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:05.424105 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:05.466973 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1369/2734] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:05 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:05.469590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a140fd-efe0-4e32-98d9-d00991e09f34 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:05.529590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:05.533178 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a140fd-efe0-4e32-98d9-d00991e09f34 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:05.538082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51d5509-e486-4099-8c12-b04d7e391695 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:05.538752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51d5509-e486-4099-8c12-b04d7e391695 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:05.539084 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51d5509-e486-4099-8c12-b04d7e391695 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:05.539389 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51d5509-e486-4099-8c12-b04d7e391695 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:05.540034 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51d5509-e486-4099-8c12-b04d7e391695 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:05.541928 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51d5509-e486-4099-8c12-b04d7e391695 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:05.615021 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1366/2735] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:57:05 2022] POST /identity/v3/projects => generated 409 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:57:05.619166 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bd1753-55ef-4b1a-81de-37af7cfde06f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:05.630946 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bd1753-55ef-4b1a-81de-37af7cfde06f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:05.686610 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c154b64e-d18d-4762-98b7-847d6feab4b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:05.687219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c154b64e-d18d-4762-98b7-847d6feab4b7 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:05.687499 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c154b64e-d18d-4762-98b7-847d6feab4b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:05.687778 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c154b64e-d18d-4762-98b7-847d6feab4b7 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:05.688402 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c154b64e-d18d-4762-98b7-847d6feab4b7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:05.690298 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c154b64e-d18d-4762-98b7-847d6feab4b7 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:05.804314 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1370/2736] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:57:05 2022] POST /identity/v3/users => generated 311 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:57:05.805667 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51d5509-e486-4099-8c12-b04d7e391695 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:05.817390 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51d5509-e486-4099-8c12-b04d7e391695 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:05.821999 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac77249f-dd89-4dd3-a5c0-3e52584dda31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:05.822621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac77249f-dd89-4dd3-a5c0-3e52584dda31 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:05.822898 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac77249f-dd89-4dd3-a5c0-3e52584dda31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:05.823194 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac77249f-dd89-4dd3-a5c0-3e52584dda31 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:05.823824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac77249f-dd89-4dd3-a5c0-3e52584dda31 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:05.825397 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac77249f-dd89-4dd3-a5c0-3e52584dda31 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:05.835588 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1367/2737] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:05 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:57:05.839918 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c154b64e-d18d-4762-98b7-847d6feab4b7 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:05.850185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c154b64e-d18d-4762-98b7-847d6feab4b7 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:05.855092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c10e5c4-61f1-4b95-862c-0028afc2b534 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:05.855761 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c10e5c4-61f1-4b95-862c-0028afc2b534 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:05.856037 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c10e5c4-61f1-4b95-862c-0028afc2b534 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:05.856940 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c10e5c4-61f1-4b95-862c-0028afc2b534 admin admin] PATH_INFO: `/v3/projects/9c3f0c2bb6144ed683268d8c99e16efb/users/1c5d9f439b274865a8cb527a6aa227b3/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:05.873017 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c10e5c4-61f1-4b95-862c-0028afc2b534 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9c3f0c2bb6144ed683268d8c99e16efb, user_id=1c5d9f439b274865a8cb527a6aa227b3, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:05.875664 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c10e5c4-61f1-4b95-862c-0028afc2b534 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:05.948939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1371/2738] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:57:05 2022] PUT /identity/v3/projects/9c3f0c2bb6144ed683268d8c99e16efb/users/1c5d9f439b274865a8cb527a6aa227b3/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:05.953079 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac77249f-dd89-4dd3-a5c0-3e52584dda31 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:05.963262 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac77249f-dd89-4dd3-a5c0-3e52584dda31 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:06.019388 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2511209d-aa09-4d49-9549-f72223d56eab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:06.019987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2511209d-aa09-4d49-9549-f72223d56eab admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:06.020265 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2511209d-aa09-4d49-9549-f72223d56eab admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:06.020542 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2511209d-aa09-4d49-9549-f72223d56eab admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:06.021285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2511209d-aa09-4d49-9549-f72223d56eab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:06.022871 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2511209d-aa09-4d49-9549-f72223d56eab admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:06.033749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1368/2739] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:05 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:57:06.038042 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c10e5c4-61f1-4b95-862c-0028afc2b534 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:06.049180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c10e5c4-61f1-4b95-862c-0028afc2b534 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:06.053776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80d28ea-2ae1-4342-9227-a5cf17b20dd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:06.054419 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80d28ea-2ae1-4342-9227-a5cf17b20dd4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:06.054701 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80d28ea-2ae1-4342-9227-a5cf17b20dd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:06.055009 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80d28ea-2ae1-4342-9227-a5cf17b20dd4 admin admin] PATH_INFO: `/v3/projects/9c3f0c2bb6144ed683268d8c99e16efb/users/1c5d9f439b274865a8cb527a6aa227b3/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:06.056977 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80d28ea-2ae1-4342-9227-a5cf17b20dd4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9c3f0c2bb6144ed683268d8c99e16efb, user_id=1c5d9f439b274865a8cb527a6aa227b3, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:06.060227 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80d28ea-2ae1-4342-9227-a5cf17b20dd4 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:06.070677 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1372/2740] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:57:06 2022] PUT /identity/v3/projects/9c3f0c2bb6144ed683268d8c99e16efb/users/1c5d9f439b274865a8cb527a6aa227b3/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:06.080810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554049f9-75a9-419e-a854-afb5ef3ed8d4 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:06.081111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554049f9-75a9-419e-a854-afb5ef3ed8d4 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:06.083914 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554049f9-75a9-419e-a854-afb5ef3ed8d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:06.115023 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-554049f9-75a9-419e-a854-afb5ef3ed8d4 None None] MFA Rules not processed for user `1c5d9f439b274865a8cb527a6aa227b3`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:06.116225 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-554049f9-75a9-419e-a854-afb5ef3ed8d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:06.186451 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1369/2741] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:06 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:06.222425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80d28ea-2ae1-4342-9227-a5cf17b20dd4 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:06.222425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c80d28ea-2ae1-4342-9227-a5cf17b20dd4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:06.276606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80d28ea-2ae1-4342-9227-a5cf17b20dd4 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:06.280878 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff509938-957e-4211-af09-265f5eefba23 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:06.281481 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff509938-957e-4211-af09-265f5eefba23 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:06.281752 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff509938-957e-4211-af09-265f5eefba23 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:06.282025 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff509938-957e-4211-af09-265f5eefba23 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:06.290721 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff509938-957e-4211-af09-265f5eefba23 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:06.292166 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff509938-957e-4211-af09-265f5eefba23 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:06.302762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1373/2742] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:06 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:57:06.743041 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554049f9-75a9-419e-a854-afb5ef3ed8d4 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:06.744423 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-554049f9-75a9-419e-a854-afb5ef3ed8d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:06.760982 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554049f9-75a9-419e-a854-afb5ef3ed8d4 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:06.764760 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a67348-ef4d-48d5-91e9-a65820f4cd3b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:06.765668 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a67348-ef4d-48d5-91e9-a65820f4cd3b service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:06.766092 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a67348-ef4d-48d5-91e9-a65820f4cd3b service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:06.766521 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a67348-ef4d-48d5-91e9-a65820f4cd3b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:06.820108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:06.829945 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a67348-ef4d-48d5-91e9-a65820f4cd3b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:06.831447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a67348-ef4d-48d5-91e9-a65820f4cd3b service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:06.841245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1370/2743] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:06 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:57:09.432561 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff509938-957e-4211-af09-265f5eefba23 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:09.434060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ff509938-957e-4211-af09-265f5eefba23 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:09.461220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:09.503142 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff509938-957e-4211-af09-265f5eefba23 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:09.508940 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d4e18d-7055-4a4d-95dd-6fc101bdf02e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:09.509577 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d4e18d-7055-4a4d-95dd-6fc101bdf02e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:09.509859 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d4e18d-7055-4a4d-95dd-6fc101bdf02e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:09.510135 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d4e18d-7055-4a4d-95dd-6fc101bdf02e admin admin] PATH_INFO: `/v3/users/198dab566d0c46278600fb0475d7a0d8` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:09.511299 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d4e18d-7055-4a4d-95dd-6fc101bdf02e admin admin] RBAC: Authorizing `identity:delete_user(user_id=198dab566d0c46278600fb0475d7a0d8)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:09.513220 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d4e18d-7055-4a4d-95dd-6fc101bdf02e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:09.736012 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-10d4e18d-7055-4a4d-95dd-6fc101bdf02e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '198dab566d0c46278600fb0475d7a0d8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:09.748021 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-10d4e18d-7055-4a4d-95dd-6fc101bdf02e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '198dab566d0c46278600fb0475d7a0d8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:09.830296 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-10d4e18d-7055-4a4d-95dd-6fc101bdf02e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '198dab566d0c46278600fb0475d7a0d8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:09.830955 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-10d4e18d-7055-4a4d-95dd-6fc101bdf02e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '198dab566d0c46278600fb0475d7a0d8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:09.838359 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-10d4e18d-7055-4a4d-95dd-6fc101bdf02e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '198dab566d0c46278600fb0475d7a0d8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:09.844849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1374/2744] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:57:09 2022] DELETE /identity/v3/users/198dab566d0c46278600fb0475d7a0d8 => generated 0 bytes in 413 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:09.849402 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d8dd3e-2ad5-42d7-995d-0ef7539d6f0d None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:09.849870 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d8dd3e-2ad5-42d7-995d-0ef7539d6f0d None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:09.849969 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d8dd3e-2ad5-42d7-995d-0ef7539d6f0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:09.869341 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-a9d8dd3e-2ad5-42d7-995d-0ef7539d6f0d None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:09.870526 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a9d8dd3e-2ad5-42d7-995d-0ef7539d6f0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:09.929956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1371/2745] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:09.944807 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d4e18d-7055-4a4d-95dd-6fc101bdf02e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:09.944807 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-10d4e18d-7055-4a4d-95dd-6fc101bdf02e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:09.998876 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d4e18d-7055-4a4d-95dd-6fc101bdf02e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:10.002422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e614e9c1-46f4-43c5-a657-3534d99662e7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:10.003055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e614e9c1-46f4-43c5-a657-3534d99662e7 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:10.003333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e614e9c1-46f4-43c5-a657-3534d99662e7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:10.003603 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e614e9c1-46f4-43c5-a657-3534d99662e7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:10.012455 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e614e9c1-46f4-43c5-a657-3534d99662e7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:10.013869 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e614e9c1-46f4-43c5-a657-3534d99662e7 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:10.023187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1375/2746] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:57:10.099607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d8dd3e-2ad5-42d7-995d-0ef7539d6f0d None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:10.101138 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a9d8dd3e-2ad5-42d7-995d-0ef7539d6f0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:10.113042 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d8dd3e-2ad5-42d7-995d-0ef7539d6f0d None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:10.122035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b875cd-342d-420e-aef3-80c874d79945 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:10.122035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b875cd-342d-420e-aef3-80c874d79945 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:10.122035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b875cd-342d-420e-aef3-80c874d79945 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:10.122035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b875cd-342d-420e-aef3-80c874d79945 admin admin] PATH_INFO: `/v3/users/c659ce1eab064353989f3ffef55705b1` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:10.122524 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b875cd-342d-420e-aef3-80c874d79945 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c659ce1eab064353989f3ffef55705b1)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:10.122524 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b875cd-342d-420e-aef3-80c874d79945 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:10.365895 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f2b875cd-342d-420e-aef3-80c874d79945 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c659ce1eab064353989f3ffef55705b1'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:10.454162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f2b875cd-342d-420e-aef3-80c874d79945 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c659ce1eab064353989f3ffef55705b1'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:10.462105 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e614e9c1-46f4-43c5-a657-3534d99662e7 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:10.463668 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e614e9c1-46f4-43c5-a657-3534d99662e7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:10.468934 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f2b875cd-342d-420e-aef3-80c874d79945 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c659ce1eab064353989f3ffef55705b1'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:10.469873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f2b875cd-342d-420e-aef3-80c874d79945 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c659ce1eab064353989f3ffef55705b1'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:10.477074 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e614e9c1-46f4-43c5-a657-3534d99662e7 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:10.483506 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a00a56e-829a-4248-896b-a740f6aa449c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:10.484066 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f2b875cd-342d-420e-aef3-80c874d79945 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c659ce1eab064353989f3ffef55705b1'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:10.484355 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a00a56e-829a-4248-896b-a740f6aa449c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:10.484809 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a00a56e-829a-4248-896b-a740f6aa449c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:10.490608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1372/2747] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:57:10 2022] DELETE /identity/v3/users/c659ce1eab064353989f3ffef55705b1 => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:10.492308 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a00a56e-829a-4248-896b-a740f6aa449c admin admin] PATH_INFO: `/v3/projects/dbfc2f845bf646cfa501228dd4ee4473` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:10.493622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a00a56e-829a-4248-896b-a740f6aa449c admin admin] RBAC: Authorizing `identity:delete_project(project_id=dbfc2f845bf646cfa501228dd4ee4473)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:10.494918 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca339308-7b42-40a5-b0f6-a94b880a723a None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:10.495214 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca339308-7b42-40a5-b0f6-a94b880a723a None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:10.495492 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca339308-7b42-40a5-b0f6-a94b880a723a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:10.504383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a00a56e-829a-4248-896b-a740f6aa449c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:10.515866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-ca339308-7b42-40a5-b0f6-a94b880a723a None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:10.517088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ca339308-7b42-40a5-b0f6-a94b880a723a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:10.577428 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1373/2748] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:10.587282 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca339308-7b42-40a5-b0f6-a94b880a723a None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:10.589101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ca339308-7b42-40a5-b0f6-a94b880a723a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:10.595043 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:10.619054 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:10.626670 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1a00a56e-829a-4248-896b-a740f6aa449c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dbfc2f845bf646cfa501228dd4ee4473', 'request_id': 'req-1a00a56e-829a-4248-896b-a740f6aa449c'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:10.633016 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1a00a56e-829a-4248-896b-a740f6aa449c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dbfc2f845bf646cfa501228dd4ee4473', 'request_id': 'req-1a00a56e-829a-4248-896b-a740f6aa449c'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:10.647454 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca339308-7b42-40a5-b0f6-a94b880a723a None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:10.694619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ab9732-5b47-475c-926a-9ba3b30f74fe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:10.695280 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ab9732-5b47-475c-926a-9ba3b30f74fe service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:10.695562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ab9732-5b47-475c-926a-9ba3b30f74fe service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:10.695835 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ab9732-5b47-475c-926a-9ba3b30f74fe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:10.750629 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1a00a56e-829a-4248-896b-a740f6aa449c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dbfc2f845bf646cfa501228dd4ee4473', 'request_id': 'req-1a00a56e-829a-4248-896b-a740f6aa449c'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:10.755117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ab9732-5b47-475c-926a-9ba3b30f74fe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:10.756575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ab9732-5b47-475c-926a-9ba3b30f74fe service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:10.762327 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1a00a56e-829a-4248-896b-a740f6aa449c admin admin] The token cache is being invalidate because project dbfc2f845bf646cfa501228dd4ee4473 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:57:10.762631 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1a00a56e-829a-4248-896b-a740f6aa449c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:10.763457 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1a00a56e-829a-4248-896b-a740f6aa449c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:10.765693 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1376/2749] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:57:10 2022] DELETE /identity/v3/projects/dbfc2f845bf646cfa501228dd4ee4473 => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:57:10.767966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1374/2750] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:57:10.770979 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a00a56e-829a-4248-896b-a740f6aa449c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:10.774822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1a00a56e-829a-4248-896b-a740f6aa449c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:10.791395 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a00a56e-829a-4248-896b-a740f6aa449c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:10.796434 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb29c19-769c-4cc8-bc61-36c899f90ae9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:10.797060 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb29c19-769c-4cc8-bc61-36c899f90ae9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:10.797338 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb29c19-769c-4cc8-bc61-36c899f90ae9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:10.797612 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb29c19-769c-4cc8-bc61-36c899f90ae9 admin admin] PATH_INFO: `/v3/users/f337fb99ff4c4e80a87127969e1021d0` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:10.798780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb29c19-769c-4cc8-bc61-36c899f90ae9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f337fb99ff4c4e80a87127969e1021d0)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:10.800777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb29c19-769c-4cc8-bc61-36c899f90ae9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:11.011674 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9fb29c19-769c-4cc8-bc61-36c899f90ae9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f337fb99ff4c4e80a87127969e1021d0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:11.023495 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9fb29c19-769c-4cc8-bc61-36c899f90ae9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f337fb99ff4c4e80a87127969e1021d0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:11.093251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9fb29c19-769c-4cc8-bc61-36c899f90ae9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f337fb99ff4c4e80a87127969e1021d0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:11.093869 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9fb29c19-769c-4cc8-bc61-36c899f90ae9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f337fb99ff4c4e80a87127969e1021d0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:11.101903 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9fb29c19-769c-4cc8-bc61-36c899f90ae9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f337fb99ff4c4e80a87127969e1021d0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:11.110034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1377/2751] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:57:10 2022] DELETE /identity/v3/users/f337fb99ff4c4e80a87127969e1021d0 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:57:11.226890 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ab9732-5b47-475c-926a-9ba3b30f74fe service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:11.228380 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a9ab9732-5b47-475c-926a-9ba3b30f74fe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:11.244511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ab9732-5b47-475c-926a-9ba3b30f74fe service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:11.313035 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d848c1-ec0a-45e4-83d1-befc51d4ac28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:11.313685 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d848c1-ec0a-45e4-83d1-befc51d4ac28 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:11.313968 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d848c1-ec0a-45e4-83d1-befc51d4ac28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:11.314241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d848c1-ec0a-45e4-83d1-befc51d4ac28 admin admin] PATH_INFO: `/v3/projects/5f35241168b542c6a61aeb5065ca80a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:11.315443 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d848c1-ec0a-45e4-83d1-befc51d4ac28 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5f35241168b542c6a61aeb5065ca80a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:11.317333 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d848c1-ec0a-45e4-83d1-befc51d4ac28 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:11.330081 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:11.465150 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c0d848c1-ec0a-45e4-83d1-befc51d4ac28 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5f35241168b542c6a61aeb5065ca80a2', 'request_id': 'req-c0d848c1-ec0a-45e4-83d1-befc51d4ac28'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:11.562725 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c0d848c1-ec0a-45e4-83d1-befc51d4ac28 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5f35241168b542c6a61aeb5065ca80a2', 'request_id': 'req-c0d848c1-ec0a-45e4-83d1-befc51d4ac28'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:11.569052 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c0d848c1-ec0a-45e4-83d1-befc51d4ac28 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5f35241168b542c6a61aeb5065ca80a2', 'request_id': 'req-c0d848c1-ec0a-45e4-83d1-befc51d4ac28'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:11.581604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c0d848c1-ec0a-45e4-83d1-befc51d4ac28 admin admin] The token cache is being invalidate because project 5f35241168b542c6a61aeb5065ca80a2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:57:11.581894 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c0d848c1-ec0a-45e4-83d1-befc51d4ac28 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:11.582523 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-c0d848c1-ec0a-45e4-83d1-befc51d4ac28 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:11.584067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1375/2752] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:57:11 2022] DELETE /identity/v3/projects/5f35241168b542c6a61aeb5065ca80a2 => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:11.588963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4035ad-f8e7-45e4-873a-dbd2ea744c67 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:11.589780 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4035ad-f8e7-45e4-873a-dbd2ea744c67 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:11.590055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4035ad-f8e7-45e4-873a-dbd2ea744c67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:11.612182 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7f4035ad-f8e7-45e4-873a-dbd2ea744c67 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:11.613377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7f4035ad-f8e7-45e4-873a-dbd2ea744c67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:11.672131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d848c1-ec0a-45e4-83d1-befc51d4ac28 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:11.672302 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1378/2753] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:57:11.678473 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c0d848c1-ec0a-45e4-83d1-befc51d4ac28 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:11.680540 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ecfc79-84d7-4b9b-97d7-cb3a4bd5cfd3 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:11.680982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ecfc79-84d7-4b9b-97d7-cb3a4bd5cfd3 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:11.681364 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ecfc79-84d7-4b9b-97d7-cb3a4bd5cfd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:11.687540 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d848c1-ec0a-45e4-83d1-befc51d4ac28 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:11.693619 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57f8155-4971-41d3-9877-a80c04645ddc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:11.694254 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57f8155-4971-41d3-9877-a80c04645ddc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:11.694648 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57f8155-4971-41d3-9877-a80c04645ddc admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:11.694944 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57f8155-4971-41d3-9877-a80c04645ddc admin admin] PATH_INFO: `/v3/projects/d2d9550c7d824227a5189f0fe585209e` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:11.696113 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57f8155-4971-41d3-9877-a80c04645ddc admin admin] RBAC: Authorizing `identity:delete_project(project_id=d2d9550c7d824227a5189f0fe585209e)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:11.698016 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57f8155-4971-41d3-9877-a80c04645ddc admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:11.706579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-10ecfc79-84d7-4b9b-97d7-cb3a4bd5cfd3 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:11.707998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-10ecfc79-84d7-4b9b-97d7-cb3a4bd5cfd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:11.717140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1379/2754] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:57:11.723356 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ecfc79-84d7-4b9b-97d7-cb3a4bd5cfd3 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:11.732454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ecfc79-84d7-4b9b-97d7-cb3a4bd5cfd3 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:11.737149 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4460fae0-e224-4833-b484-5dba85dd8ae4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:11.737743 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4460fae0-e224-4833-b484-5dba85dd8ae4 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:11.738024 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4460fae0-e224-4833-b484-5dba85dd8ae4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:11.738296 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4460fae0-e224-4833-b484-5dba85dd8ae4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:11.738966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4460fae0-e224-4833-b484-5dba85dd8ae4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:57:11.739274 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4460fae0-e224-4833-b484-5dba85dd8ae4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:11.740858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4460fae0-e224-4833-b484-5dba85dd8ae4 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:11.752264 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1380/2755] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:57:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 03:57:11.757895 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4460fae0-e224-4833-b484-5dba85dd8ae4 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:11.768135 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4460fae0-e224-4833-b484-5dba85dd8ae4 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:11.772832 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd6fc25-4770-4d89-b93e-705d699224e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:11.773403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd6fc25-4770-4d89-b93e-705d699224e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:11.773681 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd6fc25-4770-4d89-b93e-705d699224e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:11.773957 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd6fc25-4770-4d89-b93e-705d699224e7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:11.774570 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd6fc25-4770-4d89-b93e-705d699224e7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:11.776697 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd6fc25-4770-4d89-b93e-705d699224e7 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:11.831712 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f57f8155-4971-41d3-9877-a80c04645ddc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd2d9550c7d824227a5189f0fe585209e', 'request_id': 'req-f57f8155-4971-41d3-9877-a80c04645ddc'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:11.876587 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:11.886839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1381/2756] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:57:11 2022] POST /identity/v3/projects => generated 365 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:57:11.887813 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd6fc25-4770-4d89-b93e-705d699224e7 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:11.911162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f57f8155-4971-41d3-9877-a80c04645ddc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd2d9550c7d824227a5189f0fe585209e', 'request_id': 'req-f57f8155-4971-41d3-9877-a80c04645ddc'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:11.917271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f57f8155-4971-41d3-9877-a80c04645ddc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd2d9550c7d824227a5189f0fe585209e', 'request_id': 'req-f57f8155-4971-41d3-9877-a80c04645ddc'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:11.929152 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f57f8155-4971-41d3-9877-a80c04645ddc admin admin] The token cache is being invalidate because project d2d9550c7d824227a5189f0fe585209e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:57:11.929445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f57f8155-4971-41d3-9877-a80c04645ddc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:11.930192 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-f57f8155-4971-41d3-9877-a80c04645ddc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:11.931780 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1376/2757] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:57:11 2022] DELETE /identity/v3/projects/d2d9550c7d824227a5189f0fe585209e => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:57:11.936638 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57f8155-4971-41d3-9877-a80c04645ddc admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:11.938229 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-f57f8155-4971-41d3-9877-a80c04645ddc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:11.957611 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd6fc25-4770-4d89-b93e-705d699224e7 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:11.962272 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d5823a-aae8-476c-965d-215bcad98e83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:11.962877 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d5823a-aae8-476c-965d-215bcad98e83 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:11.963180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d5823a-aae8-476c-965d-215bcad98e83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:11.963452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d5823a-aae8-476c-965d-215bcad98e83 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:11.964384 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d5823a-aae8-476c-965d-215bcad98e83 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:11.965994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d5823a-aae8-476c-965d-215bcad98e83 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:12.003247 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57f8155-4971-41d3-9877-a80c04645ddc admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:12.008041 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110813b1-b7e5-4294-92ab-998d4aaa207c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:12.008661 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110813b1-b7e5-4294-92ab-998d4aaa207c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:12.008940 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110813b1-b7e5-4294-92ab-998d4aaa207c admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:12.009210 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110813b1-b7e5-4294-92ab-998d4aaa207c admin admin] PATH_INFO: `/v3/users/7bb9cca350f94c4ba5d4451db5d7bb95` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:12.010327 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110813b1-b7e5-4294-92ab-998d4aaa207c admin admin] RBAC: Authorizing `identity:delete_user(user_id=7bb9cca350f94c4ba5d4451db5d7bb95)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:12.014863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110813b1-b7e5-4294-92ab-998d4aaa207c admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:12.053682 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1382/2758] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:57:11 2022] POST /identity/v3/users => generated 283 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:57:12.057737 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d5823a-aae8-476c-965d-215bcad98e83 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:12.059635 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-60d5823a-aae8-476c-965d-215bcad98e83 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:12.071832 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d5823a-aae8-476c-965d-215bcad98e83 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:12.077287 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4e5dfa-78f8-413a-b9dc-55524a746b6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:12.077906 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e5dfa-78f8-413a-b9dc-55524a746b6b admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:12.078187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e5dfa-78f8-413a-b9dc-55524a746b6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:12.078459 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e5dfa-78f8-413a-b9dc-55524a746b6b admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:12.079153 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4e5dfa-78f8-413a-b9dc-55524a746b6b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:12.080725 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4e5dfa-78f8-413a-b9dc-55524a746b6b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:12.090779 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1383/2759] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:12 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:57:12.094648 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4e5dfa-78f8-413a-b9dc-55524a746b6b admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:12.103915 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4e5dfa-78f8-413a-b9dc-55524a746b6b admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:12.108642 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8b9271-f4ed-4a38-8405-f435355d79f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:12.109190 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8b9271-f4ed-4a38-8405-f435355d79f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:12.109472 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8b9271-f4ed-4a38-8405-f435355d79f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:12.109746 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8b9271-f4ed-4a38-8405-f435355d79f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:12.110382 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8b9271-f4ed-4a38-8405-f435355d79f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:12.111935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8b9271-f4ed-4a38-8405-f435355d79f1 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:12.122752 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1384/2760] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:12 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:57:12.127222 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8b9271-f4ed-4a38-8405-f435355d79f1 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:12.136372 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8b9271-f4ed-4a38-8405-f435355d79f1 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:12.141773 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa87b87-ec4b-4ed5-905f-8b93cfb228f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:12.142425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa87b87-ec4b-4ed5-905f-8b93cfb228f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:12.142706 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa87b87-ec4b-4ed5-905f-8b93cfb228f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:12.143018 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa87b87-ec4b-4ed5-905f-8b93cfb228f5 admin admin] PATH_INFO: `/v3/projects/72b3d7c3abda448bb208fabc93cc2ec0/users/4688254278514dda955efd2b185f37a2/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:12.145044 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=80,keystone:DELETE=18,keystone:INSERT=2 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:12.157819 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa87b87-ec4b-4ed5-905f-8b93cfb228f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=72b3d7c3abda448bb208fabc93cc2ec0, user_id=4688254278514dda955efd2b185f37a2, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:12.160504 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa87b87-ec4b-4ed5-905f-8b93cfb228f5 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:12.266417 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-110813b1-b7e5-4294-92ab-998d4aaa207c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7bb9cca350f94c4ba5d4451db5d7bb95'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:12.283330 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1385/2761] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:57:12 2022] PUT /identity/v3/projects/72b3d7c3abda448bb208fabc93cc2ec0/users/4688254278514dda955efd2b185f37a2/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:57:12.294320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2558f348-b68e-4e96-9613-87604259dae8 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:12.294610 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2558f348-b68e-4e96-9613-87604259dae8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:12.294889 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2558f348-b68e-4e96-9613-87604259dae8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:12.323998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2558f348-b68e-4e96-9613-87604259dae8 None None] MFA Rules not processed for user `4688254278514dda955efd2b185f37a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:12.325239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2558f348-b68e-4e96-9613-87604259dae8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:12.349082 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-110813b1-b7e5-4294-92ab-998d4aaa207c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7bb9cca350f94c4ba5d4451db5d7bb95'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:12.362122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-110813b1-b7e5-4294-92ab-998d4aaa207c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7bb9cca350f94c4ba5d4451db5d7bb95'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:12.362839 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-110813b1-b7e5-4294-92ab-998d4aaa207c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7bb9cca350f94c4ba5d4451db5d7bb95'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:12.370252 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-110813b1-b7e5-4294-92ab-998d4aaa207c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7bb9cca350f94c4ba5d4451db5d7bb95'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:12.377299 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1377/2762] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:57:11 2022] DELETE /identity/v3/users/7bb9cca350f94c4ba5d4451db5d7bb95 => generated 0 bytes in 441 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:57:12.410702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1386/2763] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:12 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:57:12.418500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110813b1-b7e5-4294-92ab-998d4aaa207c admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:12.421931 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-110813b1-b7e5-4294-92ab-998d4aaa207c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:12.488793 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110813b1-b7e5-4294-92ab-998d4aaa207c admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:12.492427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b415a6d7-c015-48ce-9b29-d156858ee794 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:12.493032 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b415a6d7-c015-48ce-9b29-d156858ee794 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:12.493307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b415a6d7-c015-48ce-9b29-d156858ee794 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:12.493577 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b415a6d7-c015-48ce-9b29-d156858ee794 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:12.549241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b415a6d7-c015-48ce-9b29-d156858ee794 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:12.550675 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b415a6d7-c015-48ce-9b29-d156858ee794 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:12.559333 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1378/2764] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:12 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:57:12.893909 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2558f348-b68e-4e96-9613-87604259dae8 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:12.895435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2558f348-b68e-4e96-9613-87604259dae8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:12.933877 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:12.964837 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2558f348-b68e-4e96-9613-87604259dae8 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:13.021515 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60e7cc8-0a37-439c-b9df-f7df8131b23f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:13.022160 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60e7cc8-0a37-439c-b9df-f7df8131b23f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:13.022439 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60e7cc8-0a37-439c-b9df-f7df8131b23f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:13.022714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60e7cc8-0a37-439c-b9df-f7df8131b23f admin admin] PATH_INFO: `/v3/projects/9a8c35756a5c41cda69587a6f45fa91a` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:13.023922 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d60e7cc8-0a37-439c-b9df-f7df8131b23f admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a8c35756a5c41cda69587a6f45fa91a)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:13.026024 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d60e7cc8-0a37-439c-b9df-f7df8131b23f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:13.134002 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d60e7cc8-0a37-439c-b9df-f7df8131b23f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a8c35756a5c41cda69587a6f45fa91a', 'request_id': 'req-d60e7cc8-0a37-439c-b9df-f7df8131b23f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:13.141633 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d60e7cc8-0a37-439c-b9df-f7df8131b23f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a8c35756a5c41cda69587a6f45fa91a', 'request_id': 'req-d60e7cc8-0a37-439c-b9df-f7df8131b23f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:13.249109 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d60e7cc8-0a37-439c-b9df-f7df8131b23f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a8c35756a5c41cda69587a6f45fa91a', 'request_id': 'req-d60e7cc8-0a37-439c-b9df-f7df8131b23f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:13.262090 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d60e7cc8-0a37-439c-b9df-f7df8131b23f admin admin] The token cache is being invalidate because project 9a8c35756a5c41cda69587a6f45fa91a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:57:13.262378 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d60e7cc8-0a37-439c-b9df-f7df8131b23f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:13.263057 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d60e7cc8-0a37-439c-b9df-f7df8131b23f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:13.265300 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1387/2765] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:57:12 2022] DELETE /identity/v3/projects/9a8c35756a5c41cda69587a6f45fa91a => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:13.272468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd0cf7f-6e53-4cd1-9630-18f61c348353 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:13.272953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd0cf7f-6e53-4cd1-9630-18f61c348353 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:13.273073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd0cf7f-6e53-4cd1-9630-18f61c348353 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:13.293213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-2bd0cf7f-6e53-4cd1-9630-18f61c348353 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:13.294405 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-2bd0cf7f-6e53-4cd1-9630-18f61c348353 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:13.347427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:13.356687 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1379/2766] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:13.364661 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059b6afc-fc27-4e54-ad50-ece72b3eb319 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:13.364964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059b6afc-fc27-4e54-ad50-ece72b3eb319 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:13.365238 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059b6afc-fc27-4e54-ad50-ece72b3eb319 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:13.384468 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-059b6afc-fc27-4e54-ad50-ece72b3eb319 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:13.385652 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-059b6afc-fc27-4e54-ad50-ece72b3eb319 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:13.396330 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1388/2767] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:13.398941 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd0cf7f-6e53-4cd1-9630-18f61c348353 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:13.410888 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd0cf7f-6e53-4cd1-9630-18f61c348353 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:13.418755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da90c494-f17b-4cae-8aa0-0f1604c3669e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:13.419408 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da90c494-f17b-4cae-8aa0-0f1604c3669e admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:13.419687 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da90c494-f17b-4cae-8aa0-0f1604c3669e admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:13.419960 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da90c494-f17b-4cae-8aa0-0f1604c3669e admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:13.420618 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da90c494-f17b-4cae-8aa0-0f1604c3669e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:57:13.420926 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da90c494-f17b-4cae-8aa0-0f1604c3669e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:13.422545 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da90c494-f17b-4cae-8aa0-0f1604c3669e admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:13.443217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1380/2768] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:57:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:57:13.444554 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059b6afc-fc27-4e54-ad50-ece72b3eb319 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:13.456375 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059b6afc-fc27-4e54-ad50-ece72b3eb319 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:13.461333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1990f03c-6734-41b6-9031-4add75b2180b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:13.461942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1990f03c-6734-41b6-9031-4add75b2180b admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:13.462229 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1990f03c-6734-41b6-9031-4add75b2180b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:13.462506 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1990f03c-6734-41b6-9031-4add75b2180b admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:13.463241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1990f03c-6734-41b6-9031-4add75b2180b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:13.465177 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1990f03c-6734-41b6-9031-4add75b2180b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:13.552042 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1389/2769] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:57:13 2022] POST /identity/v3/projects => generated 371 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:57:13.556297 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da90c494-f17b-4cae-8aa0-0f1604c3669e admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:13.573196 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da90c494-f17b-4cae-8aa0-0f1604c3669e admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:13.629753 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39089568-1645-4c9d-bfcf-050c2e58452c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:13.630355 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39089568-1645-4c9d-bfcf-050c2e58452c admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:13.630633 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39089568-1645-4c9d-bfcf-050c2e58452c admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:13.630953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39089568-1645-4c9d-bfcf-050c2e58452c admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:13.631589 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39089568-1645-4c9d-bfcf-050c2e58452c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:13.633511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39089568-1645-4c9d-bfcf-050c2e58452c admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:13.791925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1381/2770] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:57:13 2022] POST /identity/v3/users => generated 286 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:57:13.798105 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1990f03c-6734-41b6-9031-4add75b2180b admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:13.810560 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1990f03c-6734-41b6-9031-4add75b2180b admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:13.815719 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ddf5022-32c5-493a-8133-8e057b3b8ade admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:13.816321 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddf5022-32c5-493a-8133-8e057b3b8ade admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:13.816602 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddf5022-32c5-493a-8133-8e057b3b8ade admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:13.816874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddf5022-32c5-493a-8133-8e057b3b8ade admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:13.817778 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ddf5022-32c5-493a-8133-8e057b3b8ade admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:13.819391 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ddf5022-32c5-493a-8133-8e057b3b8ade admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:13.830053 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1390/2771] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:13 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:57:13.834288 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39089568-1645-4c9d-bfcf-050c2e58452c admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:13.848094 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39089568-1645-4c9d-bfcf-050c2e58452c admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:13.852419 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c57bdcf-a6de-40c9-8747-3dfcfebd16bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:13.853010 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c57bdcf-a6de-40c9-8747-3dfcfebd16bc admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:13.853287 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c57bdcf-a6de-40c9-8747-3dfcfebd16bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:13.853563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c57bdcf-a6de-40c9-8747-3dfcfebd16bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:13.854202 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c57bdcf-a6de-40c9-8747-3dfcfebd16bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:13.855848 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c57bdcf-a6de-40c9-8747-3dfcfebd16bc admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:13.867385 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1382/2772] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:13 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:57:13.871610 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ddf5022-32c5-493a-8133-8e057b3b8ade admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:13.882286 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ddf5022-32c5-493a-8133-8e057b3b8ade admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:13.887489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b2c7e4-6e66-4d88-a9ce-943c4d60a3db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:13.888148 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b2c7e4-6e66-4d88-a9ce-943c4d60a3db admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:13.888429 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b2c7e4-6e66-4d88-a9ce-943c4d60a3db admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:13.888703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b2c7e4-6e66-4d88-a9ce-943c4d60a3db admin admin] PATH_INFO: `/v3/projects/b47772c25b8b428dbfefba0cb62c6a96/users/81197c9d22954d388e5c80dd26bd4c60/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:13.903456 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b2c7e4-6e66-4d88-a9ce-943c4d60a3db admin admin] RBAC: Authorizing `identity:create_grant(project_id=b47772c25b8b428dbfefba0cb62c6a96, user_id=81197c9d22954d388e5c80dd26bd4c60, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:13.906251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b2c7e4-6e66-4d88-a9ce-943c4d60a3db admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:13.996506 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1391/2773] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:57:13 2022] PUT /identity/v3/projects/b47772c25b8b428dbfefba0cb62c6a96/users/81197c9d22954d388e5c80dd26bd4c60/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:14.009068 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b693b77c-df7e-49a2-aecc-fcc017b7eb63 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:14.009357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b693b77c-df7e-49a2-aecc-fcc017b7eb63 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:14.009634 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b693b77c-df7e-49a2-aecc-fcc017b7eb63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:14.038177 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-b693b77c-df7e-49a2-aecc-fcc017b7eb63 None None] MFA Rules not processed for user `81197c9d22954d388e5c80dd26bd4c60`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:14.039319 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b693b77c-df7e-49a2-aecc-fcc017b7eb63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:14.108439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1383/2774] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:14 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:14.120597 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b2c7e4-6e66-4d88-a9ce-943c4d60a3db admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:14.122294 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-72b2c7e4-6e66-4d88-a9ce-943c4d60a3db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:14.129858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:14.181141 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b2c7e4-6e66-4d88-a9ce-943c4d60a3db admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:14.185684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6b42dc-a9ad-4bd8-8c81-c1cf58813eaa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:14.186893 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6b42dc-a9ad-4bd8-8c81-c1cf58813eaa service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:14.187205 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6b42dc-a9ad-4bd8-8c81-c1cf58813eaa service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:14.187482 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6b42dc-a9ad-4bd8-8c81-c1cf58813eaa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:14.195829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6b42dc-a9ad-4bd8-8c81-c1cf58813eaa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:14.197278 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6b42dc-a9ad-4bd8-8c81-c1cf58813eaa service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:14.206286 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1392/2775] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:14 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:57:14.232781 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b693b77c-df7e-49a2-aecc-fcc017b7eb63 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:14.292899 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:14.305383 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b693b77c-df7e-49a2-aecc-fcc017b7eb63 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:14.310573 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d2c099-73c1-472a-9213-575bbe6d1f7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:14.311237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d2c099-73c1-472a-9213-575bbe6d1f7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:14.311519 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d2c099-73c1-472a-9213-575bbe6d1f7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:14.311797 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d2c099-73c1-472a-9213-575bbe6d1f7e admin admin] PATH_INFO: `/v3/users/81197c9d22954d388e5c80dd26bd4c60` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:14.312956 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d2c099-73c1-472a-9213-575bbe6d1f7e admin admin] RBAC: Authorizing `identity:delete_user(user_id=81197c9d22954d388e5c80dd26bd4c60)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:14.314856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d2c099-73c1-472a-9213-575bbe6d1f7e admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:14.490417 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-e8d2c099-73c1-472a-9213-575bbe6d1f7e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '81197c9d22954d388e5c80dd26bd4c60'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:14.596099 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-e8d2c099-73c1-472a-9213-575bbe6d1f7e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '81197c9d22954d388e5c80dd26bd4c60'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:14.609432 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-e8d2c099-73c1-472a-9213-575bbe6d1f7e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '81197c9d22954d388e5c80dd26bd4c60'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:14.610123 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-e8d2c099-73c1-472a-9213-575bbe6d1f7e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '81197c9d22954d388e5c80dd26bd4c60'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:14.617645 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-e8d2c099-73c1-472a-9213-575bbe6d1f7e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '81197c9d22954d388e5c80dd26bd4c60'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:14.623636 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1384/2776] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:57:14 2022] DELETE /identity/v3/users/81197c9d22954d388e5c80dd26bd4c60 => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:14.628003 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f81de2-ccc7-4c62-9e50-c87012533f2e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:14.628296 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f81de2-ccc7-4c62-9e50-c87012533f2e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:14.628583 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f81de2-ccc7-4c62-9e50-c87012533f2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:14.649278 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-45f81de2-ccc7-4c62-9e50-c87012533f2e None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:14.650501 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-45f81de2-ccc7-4c62-9e50-c87012533f2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:14.713095 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1393/2777] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:14.723872 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d2c099-73c1-472a-9213-575bbe6d1f7e admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:14.725259 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e8d2c099-73c1-472a-9213-575bbe6d1f7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:14.784027 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d2c099-73c1-472a-9213-575bbe6d1f7e admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:14.787676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f29023d-d126-4768-8ff4-d88fa5de0387 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:14.788277 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f29023d-d126-4768-8ff4-d88fa5de0387 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:14.788587 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f29023d-d126-4768-8ff4-d88fa5de0387 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:14.788866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f29023d-d126-4768-8ff4-d88fa5de0387 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:14.797915 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f29023d-d126-4768-8ff4-d88fa5de0387 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:14.799392 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f29023d-d126-4768-8ff4-d88fa5de0387 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:14.808889 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1385/2778] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:57:15.210302 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f81de2-ccc7-4c62-9e50-c87012533f2e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:15.218244 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-45f81de2-ccc7-4c62-9e50-c87012533f2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:15.224360 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f81de2-ccc7-4c62-9e50-c87012533f2e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:15.231096 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09345b1-b463-4c81-95e6-8af7681e735e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:15.231736 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09345b1-b463-4c81-95e6-8af7681e735e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:15.232182 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09345b1-b463-4c81-95e6-8af7681e735e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:15.232784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09345b1-b463-4c81-95e6-8af7681e735e admin admin] PATH_INFO: `/v3/projects/b47772c25b8b428dbfefba0cb62c6a96` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:15.234111 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09345b1-b463-4c81-95e6-8af7681e735e admin admin] RBAC: Authorizing `identity:delete_project(project_id=b47772c25b8b428dbfefba0cb62c6a96)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:15.236200 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09345b1-b463-4c81-95e6-8af7681e735e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:15.349013 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f09345b1-b463-4c81-95e6-8af7681e735e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b47772c25b8b428dbfefba0cb62c6a96', 'request_id': 'req-f09345b1-b463-4c81-95e6-8af7681e735e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:15.355601 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f09345b1-b463-4c81-95e6-8af7681e735e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b47772c25b8b428dbfefba0cb62c6a96', 'request_id': 'req-f09345b1-b463-4c81-95e6-8af7681e735e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:15.457493 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f09345b1-b463-4c81-95e6-8af7681e735e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b47772c25b8b428dbfefba0cb62c6a96', 'request_id': 'req-f09345b1-b463-4c81-95e6-8af7681e735e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:15.468713 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f09345b1-b463-4c81-95e6-8af7681e735e admin admin] The token cache is being invalidate because project b47772c25b8b428dbfefba0cb62c6a96 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:57:15.469012 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f09345b1-b463-4c81-95e6-8af7681e735e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:15.469661 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f09345b1-b463-4c81-95e6-8af7681e735e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:15.471233 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1394/2779] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:57:15 2022] DELETE /identity/v3/projects/b47772c25b8b428dbfefba0cb62c6a96 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:17.251505 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f29023d-d126-4768-8ff4-d88fa5de0387 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:17.252942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-2f29023d-d126-4768-8ff4-d88fa5de0387 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:17.265945 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f29023d-d126-4768-8ff4-d88fa5de0387 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:17.322339 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d0de0f-c3c8-4469-a258-fdc834ed30d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:17.323000 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d0de0f-c3c8-4469-a258-fdc834ed30d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:17.323281 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d0de0f-c3c8-4469-a258-fdc834ed30d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:17.323553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d0de0f-c3c8-4469-a258-fdc834ed30d8 admin admin] PATH_INFO: `/v3/users/5dcbee68f48f4b348954b6edc6d21b8f` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:17.324682 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d0de0f-c3c8-4469-a258-fdc834ed30d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5dcbee68f48f4b348954b6edc6d21b8f)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:17.326608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d0de0f-c3c8-4469-a258-fdc834ed30d8 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:17.348014 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:17.582937 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a9d0de0f-c3c8-4469-a258-fdc834ed30d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5dcbee68f48f4b348954b6edc6d21b8f'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:17.662132 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a9d0de0f-c3c8-4469-a258-fdc834ed30d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5dcbee68f48f4b348954b6edc6d21b8f'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:17.674894 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a9d0de0f-c3c8-4469-a258-fdc834ed30d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5dcbee68f48f4b348954b6edc6d21b8f'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:17.675601 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a9d0de0f-c3c8-4469-a258-fdc834ed30d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5dcbee68f48f4b348954b6edc6d21b8f'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:17.682623 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-a9d0de0f-c3c8-4469-a258-fdc834ed30d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5dcbee68f48f4b348954b6edc6d21b8f'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:17.688909 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1386/2780] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:57:17 2022] DELETE /identity/v3/users/5dcbee68f48f4b348954b6edc6d21b8f => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:17.693576 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f7746f-ce14-4204-883c-d55946f98972 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:17.694075 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f7746f-ce14-4204-883c-d55946f98972 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:17.694187 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f7746f-ce14-4204-883c-d55946f98972 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:17.713824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e9f7746f-ce14-4204-883c-d55946f98972 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:17.715058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e9f7746f-ce14-4204-883c-d55946f98972 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:17.741007 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:17.773600 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1395/2781] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:17.784133 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d0de0f-c3c8-4469-a258-fdc834ed30d8 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:17.785679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a9d0de0f-c3c8-4469-a258-fdc834ed30d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:17.846183 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d0de0f-c3c8-4469-a258-fdc834ed30d8 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:17.849899 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02664a3a-be41-4287-b4db-a445f018a953 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:17.850578 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02664a3a-be41-4287-b4db-a445f018a953 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:17.850860 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02664a3a-be41-4287-b4db-a445f018a953 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:17.851177 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02664a3a-be41-4287-b4db-a445f018a953 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:17.871296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02664a3a-be41-4287-b4db-a445f018a953 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:17.871296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02664a3a-be41-4287-b4db-a445f018a953 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:17.871729 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1387/2782] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:57:18.585055 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f7746f-ce14-4204-883c-d55946f98972 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:18.589987 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e9f7746f-ce14-4204-883c-d55946f98972 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:18.599939 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f7746f-ce14-4204-883c-d55946f98972 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:18.604834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8688a5-7d4b-455b-8d21-a8d1901e0c8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:18.605468 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8688a5-7d4b-455b-8d21-a8d1901e0c8d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:18.605744 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8688a5-7d4b-455b-8d21-a8d1901e0c8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:18.606122 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8688a5-7d4b-455b-8d21-a8d1901e0c8d admin admin] PATH_INFO: `/v3/projects/df64a64f6a9a4f78b77c8b0e3b982a6b` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:18.607370 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8688a5-7d4b-455b-8d21-a8d1901e0c8d admin admin] RBAC: Authorizing `identity:delete_project(project_id=df64a64f6a9a4f78b77c8b0e3b982a6b)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:18.610072 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8688a5-7d4b-455b-8d21-a8d1901e0c8d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:18.808331 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4f8688a5-7d4b-455b-8d21-a8d1901e0c8d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df64a64f6a9a4f78b77c8b0e3b982a6b', 'request_id': 'req-4f8688a5-7d4b-455b-8d21-a8d1901e0c8d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:18.815017 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4f8688a5-7d4b-455b-8d21-a8d1901e0c8d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df64a64f6a9a4f78b77c8b0e3b982a6b', 'request_id': 'req-4f8688a5-7d4b-455b-8d21-a8d1901e0c8d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:18.914414 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4f8688a5-7d4b-455b-8d21-a8d1901e0c8d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df64a64f6a9a4f78b77c8b0e3b982a6b', 'request_id': 'req-4f8688a5-7d4b-455b-8d21-a8d1901e0c8d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:18.926172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4f8688a5-7d4b-455b-8d21-a8d1901e0c8d admin admin] The token cache is being invalidate because project df64a64f6a9a4f78b77c8b0e3b982a6b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:57:18.926460 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4f8688a5-7d4b-455b-8d21-a8d1901e0c8d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:18.927129 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4f8688a5-7d4b-455b-8d21-a8d1901e0c8d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:18.928653 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1396/2783] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:57:18 2022] DELETE /identity/v3/projects/df64a64f6a9a4f78b77c8b0e3b982a6b => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:18.932638 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02664a3a-be41-4287-b4db-a445f018a953 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:18.943584 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02664a3a-be41-4287-b4db-a445f018a953 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:19.001113 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af62270-295b-484f-a931-3ec81891b5b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:19.001738 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af62270-295b-484f-a931-3ec81891b5b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:19.002017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af62270-295b-484f-a931-3ec81891b5b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:19.002284 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af62270-295b-484f-a931-3ec81891b5b2 admin admin] PATH_INFO: `/v3/users/1c5d9f439b274865a8cb527a6aa227b3` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:19.003418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af62270-295b-484f-a931-3ec81891b5b2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1c5d9f439b274865a8cb527a6aa227b3)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:19.005310 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af62270-295b-484f-a931-3ec81891b5b2 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:19.221838 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:DELETE=14,keystone:SELECT=85,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:19.227958 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1af62270-295b-484f-a931-3ec81891b5b2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1c5d9f439b274865a8cb527a6aa227b3'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:19.344592 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1af62270-295b-484f-a931-3ec81891b5b2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1c5d9f439b274865a8cb527a6aa227b3'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:19.357230 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1af62270-295b-484f-a931-3ec81891b5b2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1c5d9f439b274865a8cb527a6aa227b3'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:19.358371 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1af62270-295b-484f-a931-3ec81891b5b2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1c5d9f439b274865a8cb527a6aa227b3'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:19.365122 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1af62270-295b-484f-a931-3ec81891b5b2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1c5d9f439b274865a8cb527a6aa227b3'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:19.370845 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1388/2784] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:57:18 2022] DELETE /identity/v3/users/1c5d9f439b274865a8cb527a6aa227b3 => generated 0 bytes in 439 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:19.821176 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8688a5-7d4b-455b-8d21-a8d1901e0c8d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:19.825907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4f8688a5-7d4b-455b-8d21-a8d1901e0c8d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:19.844967 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8688a5-7d4b-455b-8d21-a8d1901e0c8d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:19.901617 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3830d2b3-c4c0-4cff-9970-b8841b6f3cc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:19.902249 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3830d2b3-c4c0-4cff-9970-b8841b6f3cc9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:19.902530 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3830d2b3-c4c0-4cff-9970-b8841b6f3cc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:19.902804 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3830d2b3-c4c0-4cff-9970-b8841b6f3cc9 admin admin] PATH_INFO: `/v3/projects/9c3f0c2bb6144ed683268d8c99e16efb` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:19.903959 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3830d2b3-c4c0-4cff-9970-b8841b6f3cc9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9c3f0c2bb6144ed683268d8c99e16efb)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:19.905993 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3830d2b3-c4c0-4cff-9970-b8841b6f3cc9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:20.024282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3830d2b3-c4c0-4cff-9970-b8841b6f3cc9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9c3f0c2bb6144ed683268d8c99e16efb', 'request_id': 'req-3830d2b3-c4c0-4cff-9970-b8841b6f3cc9'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:20.030709 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3830d2b3-c4c0-4cff-9970-b8841b6f3cc9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9c3f0c2bb6144ed683268d8c99e16efb', 'request_id': 'req-3830d2b3-c4c0-4cff-9970-b8841b6f3cc9'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:20.033360 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:20.093239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3830d2b3-c4c0-4cff-9970-b8841b6f3cc9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9c3f0c2bb6144ed683268d8c99e16efb', 'request_id': 'req-3830d2b3-c4c0-4cff-9970-b8841b6f3cc9'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:20.103956 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3830d2b3-c4c0-4cff-9970-b8841b6f3cc9 admin admin] The token cache is being invalidate because project 9c3f0c2bb6144ed683268d8c99e16efb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:57:20.104239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3830d2b3-c4c0-4cff-9970-b8841b6f3cc9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:20.104854 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3830d2b3-c4c0-4cff-9970-b8841b6f3cc9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:20.106320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1397/2785] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:57:19 2022] DELETE /identity/v3/projects/9c3f0c2bb6144ed683268d8c99e16efb => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:20.112065 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ae92e5-4d3d-4243-836f-c9894a236bc0 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:20.112529 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ae92e5-4d3d-4243-836f-c9894a236bc0 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:20.112652 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ae92e5-4d3d-4243-836f-c9894a236bc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:20.133183 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-a7ae92e5-4d3d-4243-836f-c9894a236bc0 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:20.134374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a7ae92e5-4d3d-4243-836f-c9894a236bc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:20.193640 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1389/2786] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:20.201899 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac0fa41-febb-43b3-b489-831459a2912c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:20.202320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac0fa41-febb-43b3-b489-831459a2912c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:20.202595 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac0fa41-febb-43b3-b489-831459a2912c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:20.223166 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-cac0fa41-febb-43b3-b489-831459a2912c None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:20.224384 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cac0fa41-febb-43b3-b489-831459a2912c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:20.232308 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1398/2787] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:57:20.237814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ae92e5-4d3d-4243-836f-c9894a236bc0 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:20.247567 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ae92e5-4d3d-4243-836f-c9894a236bc0 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:20.252391 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42c2665-350b-45a4-8847-dbd304fe07ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:20.252985 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42c2665-350b-45a4-8847-dbd304fe07ae admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:20.253267 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42c2665-350b-45a4-8847-dbd304fe07ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:20.253541 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42c2665-350b-45a4-8847-dbd304fe07ae admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:20.254213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42c2665-350b-45a4-8847-dbd304fe07ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:57:20.254518 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42c2665-350b-45a4-8847-dbd304fe07ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:20.256203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42c2665-350b-45a4-8847-dbd304fe07ae admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:20.267879 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1390/2788] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:57:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:57:20.272269 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac0fa41-febb-43b3-b489-831459a2912c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:20.290123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac0fa41-febb-43b3-b489-831459a2912c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:20.290123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a9d97b-f7c2-4222-aa7b-9b214110d1ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:20.290123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a9d97b-f7c2-4222-aa7b-9b214110d1ba admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:20.290123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a9d97b-f7c2-4222-aa7b-9b214110d1ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:20.290593 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a9d97b-f7c2-4222-aa7b-9b214110d1ba admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:20.290593 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a9d97b-f7c2-4222-aa7b-9b214110d1ba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:20.290593 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a9d97b-f7c2-4222-aa7b-9b214110d1ba admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:20.414820 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1399/2789] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:57:20 2022] POST /identity/v3/projects => generated 375 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:57:20.419225 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42c2665-350b-45a4-8847-dbd304fe07ae admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:20.429885 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42c2665-350b-45a4-8847-dbd304fe07ae admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:20.487822 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d1f3db-9177-4032-a989-37e0b916e437 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:20.488437 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d1f3db-9177-4032-a989-37e0b916e437 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:20.488725 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d1f3db-9177-4032-a989-37e0b916e437 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:20.489002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d1f3db-9177-4032-a989-37e0b916e437 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:20.489656 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d1f3db-9177-4032-a989-37e0b916e437 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:20.491716 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d1f3db-9177-4032-a989-37e0b916e437 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:20.608079 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1391/2790] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:57:20 2022] POST /identity/v3/users => generated 288 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:57:20.612233 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a9d97b-f7c2-4222-aa7b-9b214110d1ba admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:20.622820 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a9d97b-f7c2-4222-aa7b-9b214110d1ba admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:20.628433 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f0b046-4064-4492-966b-2ac52e6e81d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:20.635104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f0b046-4064-4492-966b-2ac52e6e81d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:20.635104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f0b046-4064-4492-966b-2ac52e6e81d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:20.635104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f0b046-4064-4492-966b-2ac52e6e81d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:20.635104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f0b046-4064-4492-966b-2ac52e6e81d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:20.635104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f0b046-4064-4492-966b-2ac52e6e81d2 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:20.643554 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1400/2791] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:20 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:57:20.647562 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d1f3db-9177-4032-a989-37e0b916e437 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:20.658468 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d1f3db-9177-4032-a989-37e0b916e437 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:20.663302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f685842-44f2-4254-84a2-41d776907ff1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:20.663887 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f685842-44f2-4254-84a2-41d776907ff1 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:20.664168 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f685842-44f2-4254-84a2-41d776907ff1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:20.664442 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f685842-44f2-4254-84a2-41d776907ff1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:20.665077 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f685842-44f2-4254-84a2-41d776907ff1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:20.666693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f685842-44f2-4254-84a2-41d776907ff1 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:20.678118 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1392/2792] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:57:20.682342 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f0b046-4064-4492-966b-2ac52e6e81d2 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:20.693243 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f0b046-4064-4492-966b-2ac52e6e81d2 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:20.698021 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d74cfe-84b1-47f1-9fcc-3af6b0f26390 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:20.698815 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d74cfe-84b1-47f1-9fcc-3af6b0f26390 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:20.699185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d74cfe-84b1-47f1-9fcc-3af6b0f26390 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:20.699532 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d74cfe-84b1-47f1-9fcc-3af6b0f26390 admin admin] PATH_INFO: `/v3/projects/ce272bebfd5a4a2c9439e9ebe4592414/users/eda9272c2929423aa16f6242337cba8e/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:20.715537 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d74cfe-84b1-47f1-9fcc-3af6b0f26390 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ce272bebfd5a4a2c9439e9ebe4592414, user_id=eda9272c2929423aa16f6242337cba8e, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:20.719149 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d74cfe-84b1-47f1-9fcc-3af6b0f26390 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:20.795517 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1401/2793] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:57:20 2022] PUT /identity/v3/projects/ce272bebfd5a4a2c9439e9ebe4592414/users/eda9272c2929423aa16f6242337cba8e/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:57:20.808254 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8159fd9-1e96-487f-a2e2-49d1f1eadffc None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:20.808540 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8159fd9-1e96-487f-a2e2-49d1f1eadffc None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:20.808814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8159fd9-1e96-487f-a2e2-49d1f1eadffc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:20.841933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-a8159fd9-1e96-487f-a2e2-49d1f1eadffc None None] MFA Rules not processed for user `eda9272c2929423aa16f6242337cba8e`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:20.843935 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a8159fd9-1e96-487f-a2e2-49d1f1eadffc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:20.852826 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:DELETE=5,keystone:SELECT=91,keystone:INSERT=4 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:20.905912 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1393/2794] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:20 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:20.910288 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d74cfe-84b1-47f1-9fcc-3af6b0f26390 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:20.990940 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d74cfe-84b1-47f1-9fcc-3af6b0f26390 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:20.997197 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920c0e59-a9ee-4ccc-a116-10fb9601dae2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:20.997801 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920c0e59-a9ee-4ccc-a116-10fb9601dae2 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:20.998092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920c0e59-a9ee-4ccc-a116-10fb9601dae2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:20.999016 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920c0e59-a9ee-4ccc-a116-10fb9601dae2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:20.999642 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920c0e59-a9ee-4ccc-a116-10fb9601dae2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:21.001612 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920c0e59-a9ee-4ccc-a116-10fb9601dae2 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:21.115202 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1402/2795] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:57:20 2022] POST /identity/v3/projects => generated 375 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:57:21.116349 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8159fd9-1e96-487f-a2e2-49d1f1eadffc None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:21.127687 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8159fd9-1e96-487f-a2e2-49d1f1eadffc None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:21.184333 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae452dca-386e-49f8-9a41-07fd146b3994 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:21.185256 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae452dca-386e-49f8-9a41-07fd146b3994 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:21.185541 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae452dca-386e-49f8-9a41-07fd146b3994 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:21.186319 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae452dca-386e-49f8-9a41-07fd146b3994 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:21.186984 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae452dca-386e-49f8-9a41-07fd146b3994 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:21.188955 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae452dca-386e-49f8-9a41-07fd146b3994 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:21.293141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1394/2796] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:57:21 2022] POST /identity/v3/users => generated 294 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:57:21.297190 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920c0e59-a9ee-4ccc-a116-10fb9601dae2 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:21.307629 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920c0e59-a9ee-4ccc-a116-10fb9601dae2 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:21.314170 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52275edf-58f4-4d9f-8e9d-589c4028c166 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:21.315188 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52275edf-58f4-4d9f-8e9d-589c4028c166 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:21.318866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52275edf-58f4-4d9f-8e9d-589c4028c166 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:21.318866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52275edf-58f4-4d9f-8e9d-589c4028c166 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:21.318866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52275edf-58f4-4d9f-8e9d-589c4028c166 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:21.318866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52275edf-58f4-4d9f-8e9d-589c4028c166 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:21.334523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1403/2797] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:21 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:57:21.338942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae452dca-386e-49f8-9a41-07fd146b3994 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:21.349933 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae452dca-386e-49f8-9a41-07fd146b3994 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:21.356360 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baead386-331f-4316-a3f6-8995a7839361 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:21.357117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baead386-331f-4316-a3f6-8995a7839361 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:21.357486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baead386-331f-4316-a3f6-8995a7839361 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:21.358370 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baead386-331f-4316-a3f6-8995a7839361 admin admin] PATH_INFO: `/v3/projects/460d7af798254ee0a6317a289badb677/users/57896da6163c4cb8b6b63f022eb9ba26/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:21.374440 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baead386-331f-4316-a3f6-8995a7839361 admin admin] RBAC: Authorizing `identity:create_grant(project_id=460d7af798254ee0a6317a289badb677, user_id=57896da6163c4cb8b6b63f022eb9ba26, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:21.377313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baead386-331f-4316-a3f6-8995a7839361 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:21.464381 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1395/2798] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:57:21 2022] PUT /identity/v3/projects/460d7af798254ee0a6317a289badb677/users/57896da6163c4cb8b6b63f022eb9ba26/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:57:21.468646 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52275edf-58f4-4d9f-8e9d-589c4028c166 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:21.479862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52275edf-58f4-4d9f-8e9d-589c4028c166 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:21.529945 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:DELETE=3,keystone:SELECT=94,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:21.536862 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9877c1d1-b13c-4869-b045-b80ec4250e7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:21.537470 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9877c1d1-b13c-4869-b045-b80ec4250e7d admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:21.537755 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9877c1d1-b13c-4869-b045-b80ec4250e7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:21.538025 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9877c1d1-b13c-4869-b045-b80ec4250e7d admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:21.538659 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9877c1d1-b13c-4869-b045-b80ec4250e7d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:21.540282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9877c1d1-b13c-4869-b045-b80ec4250e7d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:21.550293 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1404/2799] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:21 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:57:21.554424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baead386-331f-4316-a3f6-8995a7839361 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:21.565499 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baead386-331f-4316-a3f6-8995a7839361 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:21.571608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1cf55c-58af-46ca-8b7f-3e7608f171b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:21.572336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1cf55c-58af-46ca-8b7f-3e7608f171b5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:21.572683 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1cf55c-58af-46ca-8b7f-3e7608f171b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:21.573021 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1cf55c-58af-46ca-8b7f-3e7608f171b5 admin admin] PATH_INFO: `/v3/projects/460d7af798254ee0a6317a289badb677/users/57896da6163c4cb8b6b63f022eb9ba26/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:21.576140 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1cf55c-58af-46ca-8b7f-3e7608f171b5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=460d7af798254ee0a6317a289badb677, user_id=57896da6163c4cb8b6b63f022eb9ba26, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:21.579604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1cf55c-58af-46ca-8b7f-3e7608f171b5 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:21.591109 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1396/2800] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:57:21 2022] PUT /identity/v3/projects/460d7af798254ee0a6317a289badb677/users/57896da6163c4cb8b6b63f022eb9ba26/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:21.662732 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680a754c-2cc7-42ee-990b-22de5e33c700 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:21.663172 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680a754c-2cc7-42ee-990b-22de5e33c700 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:21.663423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680a754c-2cc7-42ee-990b-22de5e33c700 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:21.693347 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-680a754c-2cc7-42ee-990b-22de5e33c700 None None] MFA Rules not processed for user `57896da6163c4cb8b6b63f022eb9ba26`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:21.693347 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-680a754c-2cc7-42ee-990b-22de5e33c700 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:21.763968 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1405/2801] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:21 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:21.775980 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1cf55c-58af-46ca-8b7f-3e7608f171b5 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:21.777791 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-9e1cf55c-58af-46ca-8b7f-3e7608f171b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:21.839294 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1cf55c-58af-46ca-8b7f-3e7608f171b5 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:21.843582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083228db-18e1-4c7b-b7c2-b2144359deba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:21.844272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083228db-18e1-4c7b-b7c2-b2144359deba service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:21.844467 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083228db-18e1-4c7b-b7c2-b2144359deba service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:21.844739 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083228db-18e1-4c7b-b7c2-b2144359deba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:21.862299 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:21.902371 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083228db-18e1-4c7b-b7c2-b2144359deba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:21.903840 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083228db-18e1-4c7b-b7c2-b2144359deba service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:21.912839 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1397/2802] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:21 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:57:27.004581 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680a754c-2cc7-42ee-990b-22de5e33c700 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:27.067029 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680a754c-2cc7-42ee-990b-22de5e33c700 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:27.072510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206725c0-ad28-4407-b1e5-7532aa9c3176 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:27.073628 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206725c0-ad28-4407-b1e5-7532aa9c3176 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:27.074250 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206725c0-ad28-4407-b1e5-7532aa9c3176 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:27.074702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206725c0-ad28-4407-b1e5-7532aa9c3176 admin admin] PATH_INFO: `/v3/users/eda9272c2929423aa16f6242337cba8e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:27.076733 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206725c0-ad28-4407-b1e5-7532aa9c3176 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eda9272c2929423aa16f6242337cba8e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:27.079241 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206725c0-ad28-4407-b1e5-7532aa9c3176 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:27.325604 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-206725c0-ad28-4407-b1e5-7532aa9c3176 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eda9272c2929423aa16f6242337cba8e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:27.337876 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-206725c0-ad28-4407-b1e5-7532aa9c3176 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eda9272c2929423aa16f6242337cba8e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:27.418446 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-206725c0-ad28-4407-b1e5-7532aa9c3176 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eda9272c2929423aa16f6242337cba8e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:27.419117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-206725c0-ad28-4407-b1e5-7532aa9c3176 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eda9272c2929423aa16f6242337cba8e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:27.425180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:27.427178 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-206725c0-ad28-4407-b1e5-7532aa9c3176 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eda9272c2929423aa16f6242337cba8e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:27.433021 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1406/2803] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:57:27 2022] DELETE /identity/v3/users/eda9272c2929423aa16f6242337cba8e => generated 0 bytes in 429 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:27.437859 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f273d8-02b7-44bf-bbba-e5d525fb4f02 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:27.438316 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f273d8-02b7-44bf-bbba-e5d525fb4f02 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:27.438415 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f273d8-02b7-44bf-bbba-e5d525fb4f02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:27.458172 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-97f273d8-02b7-44bf-bbba-e5d525fb4f02 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:27.459428 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-97f273d8-02b7-44bf-bbba-e5d525fb4f02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:27.518298 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1398/2804] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:27.527879 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206725c0-ad28-4407-b1e5-7532aa9c3176 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:27.529283 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-206725c0-ad28-4407-b1e5-7532aa9c3176 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:27.587527 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206725c0-ad28-4407-b1e5-7532aa9c3176 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:27.591325 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db6bb86-d2a1-4b5e-a392-170d256d36b5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:27.591987 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db6bb86-d2a1-4b5e-a392-170d256d36b5 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:27.592272 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db6bb86-d2a1-4b5e-a392-170d256d36b5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:27.592553 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db6bb86-d2a1-4b5e-a392-170d256d36b5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:27.601870 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db6bb86-d2a1-4b5e-a392-170d256d36b5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:27.603341 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db6bb86-d2a1-4b5e-a392-170d256d36b5 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:27.613607 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1407/2805] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:57:28.067801 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f273d8-02b7-44bf-bbba-e5d525fb4f02 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:28.069285 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-97f273d8-02b7-44bf-bbba-e5d525fb4f02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:28.081626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f273d8-02b7-44bf-bbba-e5d525fb4f02 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:28.099447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042da383-45cf-40bd-bb7c-e5f05d9e4204 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:28.100102 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042da383-45cf-40bd-bb7c-e5f05d9e4204 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:28.102018 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042da383-45cf-40bd-bb7c-e5f05d9e4204 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:28.102018 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042da383-45cf-40bd-bb7c-e5f05d9e4204 admin admin] PATH_INFO: `/v3/projects/ce272bebfd5a4a2c9439e9ebe4592414` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:28.102018 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042da383-45cf-40bd-bb7c-e5f05d9e4204 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ce272bebfd5a4a2c9439e9ebe4592414)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:28.103799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042da383-45cf-40bd-bb7c-e5f05d9e4204 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:28.230967 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-042da383-45cf-40bd-bb7c-e5f05d9e4204 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ce272bebfd5a4a2c9439e9ebe4592414', 'request_id': 'req-042da383-45cf-40bd-bb7c-e5f05d9e4204'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:28.321411 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-042da383-45cf-40bd-bb7c-e5f05d9e4204 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ce272bebfd5a4a2c9439e9ebe4592414', 'request_id': 'req-042da383-45cf-40bd-bb7c-e5f05d9e4204'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:28.327809 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-042da383-45cf-40bd-bb7c-e5f05d9e4204 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ce272bebfd5a4a2c9439e9ebe4592414', 'request_id': 'req-042da383-45cf-40bd-bb7c-e5f05d9e4204'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:28.339324 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-042da383-45cf-40bd-bb7c-e5f05d9e4204 admin admin] The token cache is being invalidate because project ce272bebfd5a4a2c9439e9ebe4592414 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:57:28.339617 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-042da383-45cf-40bd-bb7c-e5f05d9e4204 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:28.340265 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-042da383-45cf-40bd-bb7c-e5f05d9e4204 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:28.341879 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1399/2806] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:57:28 2022] DELETE /identity/v3/projects/ce272bebfd5a4a2c9439e9ebe4592414 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:28.345885 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db6bb86-d2a1-4b5e-a392-170d256d36b5 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:28.356118 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db6bb86-d2a1-4b5e-a392-170d256d36b5 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:28.411080 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c795f09-4737-4d64-b7f1-8d6aa19e2899 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:28.411705 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c795f09-4737-4d64-b7f1-8d6aa19e2899 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:28.411980 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c795f09-4737-4d64-b7f1-8d6aa19e2899 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:28.412306 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c795f09-4737-4d64-b7f1-8d6aa19e2899 admin admin] PATH_INFO: `/v3/users/57896da6163c4cb8b6b63f022eb9ba26` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:28.413403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c795f09-4737-4d64-b7f1-8d6aa19e2899 admin admin] RBAC: Authorizing `identity:delete_user(user_id=57896da6163c4cb8b6b63f022eb9ba26)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:28.415375 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c795f09-4737-4d64-b7f1-8d6aa19e2899 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:28.593140 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1c795f09-4737-4d64-b7f1-8d6aa19e2899 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '57896da6163c4cb8b6b63f022eb9ba26'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:28.605778 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1c795f09-4737-4d64-b7f1-8d6aa19e2899 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '57896da6163c4cb8b6b63f022eb9ba26'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:28.681849 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1c795f09-4737-4d64-b7f1-8d6aa19e2899 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '57896da6163c4cb8b6b63f022eb9ba26'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:28.682460 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1c795f09-4737-4d64-b7f1-8d6aa19e2899 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '57896da6163c4cb8b6b63f022eb9ba26'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:28.690874 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1c795f09-4737-4d64-b7f1-8d6aa19e2899 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '57896da6163c4cb8b6b63f022eb9ba26'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:28.696788 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1408/2807] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:57:28 2022] DELETE /identity/v3/users/57896da6163c4cb8b6b63f022eb9ba26 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:29.171320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042da383-45cf-40bd-bb7c-e5f05d9e4204 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:29.172894 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-042da383-45cf-40bd-bb7c-e5f05d9e4204 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:29.186582 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042da383-45cf-40bd-bb7c-e5f05d9e4204 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:29.228724 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:29.244796 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22c7ebd-2e03-423b-8aa0-02f509762584 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:29.245424 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22c7ebd-2e03-423b-8aa0-02f509762584 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:29.245705 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22c7ebd-2e03-423b-8aa0-02f509762584 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:29.245982 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22c7ebd-2e03-423b-8aa0-02f509762584 admin admin] PATH_INFO: `/v3/projects/460d7af798254ee0a6317a289badb677` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:29.247251 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22c7ebd-2e03-423b-8aa0-02f509762584 admin admin] RBAC: Authorizing `identity:delete_project(project_id=460d7af798254ee0a6317a289badb677)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:29.249365 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22c7ebd-2e03-423b-8aa0-02f509762584 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:29.387542 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-e22c7ebd-2e03-423b-8aa0-02f509762584 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '460d7af798254ee0a6317a289badb677', 'request_id': 'req-e22c7ebd-2e03-423b-8aa0-02f509762584'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:29.469905 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-e22c7ebd-2e03-423b-8aa0-02f509762584 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '460d7af798254ee0a6317a289badb677', 'request_id': 'req-e22c7ebd-2e03-423b-8aa0-02f509762584'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:29.476701 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-e22c7ebd-2e03-423b-8aa0-02f509762584 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '460d7af798254ee0a6317a289badb677', 'request_id': 'req-e22c7ebd-2e03-423b-8aa0-02f509762584'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:29.489191 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-e22c7ebd-2e03-423b-8aa0-02f509762584 admin admin] The token cache is being invalidate because project 460d7af798254ee0a6317a289badb677 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:57:29.489484 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-e22c7ebd-2e03-423b-8aa0-02f509762584 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:29.490117 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-e22c7ebd-2e03-423b-8aa0-02f509762584 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:29.491824 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1400/2808] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:57:29 2022] DELETE /identity/v3/projects/460d7af798254ee0a6317a289badb677 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:29.497052 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d955a62-28df-4253-8ddd-f04568312af9 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:29.497711 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d955a62-28df-4253-8ddd-f04568312af9 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:29.497711 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d955a62-28df-4253-8ddd-f04568312af9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:29.518066 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-5d955a62-28df-4253-8ddd-f04568312af9 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:29.519330 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5d955a62-28df-4253-8ddd-f04568312af9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:29.531086 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:29.578703 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1409/2809] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:29.586702 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08f23d1-6809-4779-a740-4cbc07f71540 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:29.587029 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08f23d1-6809-4779-a740-4cbc07f71540 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:29.587309 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08f23d1-6809-4779-a740-4cbc07f71540 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:29.608478 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-e08f23d1-6809-4779-a740-4cbc07f71540 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:29.609725 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e08f23d1-6809-4779-a740-4cbc07f71540 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:29.618829 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1401/2810] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:29.623111 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d955a62-28df-4253-8ddd-f04568312af9 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:29.632930 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d955a62-28df-4253-8ddd-f04568312af9 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:29.637633 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9d07fd-5f12-4c69-8bb4-a1775a06b6d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:29.638234 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9d07fd-5f12-4c69-8bb4-a1775a06b6d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:29.638511 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9d07fd-5f12-4c69-8bb4-a1775a06b6d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:29.638784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9d07fd-5f12-4c69-8bb4-a1775a06b6d6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:29.639471 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9d07fd-5f12-4c69-8bb4-a1775a06b6d6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:57:29.639895 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9d07fd-5f12-4c69-8bb4-a1775a06b6d6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:29.641467 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9d07fd-5f12-4c69-8bb4-a1775a06b6d6 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:29.653329 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1410/2811] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:57:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:57:29.657644 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08f23d1-6809-4779-a740-4cbc07f71540 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:29.667871 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08f23d1-6809-4779-a740-4cbc07f71540 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:29.672678 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517b10dd-5e73-4f6b-8988-111b90da9380 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:29.673296 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517b10dd-5e73-4f6b-8988-111b90da9380 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:29.673578 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517b10dd-5e73-4f6b-8988-111b90da9380 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:29.673880 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517b10dd-5e73-4f6b-8988-111b90da9380 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:29.674529 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517b10dd-5e73-4f6b-8988-111b90da9380 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:29.676575 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517b10dd-5e73-4f6b-8988-111b90da9380 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:29.791762 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1402/2812] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:57:29 2022] POST /identity/v3/projects => generated 383 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:57:29.795907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9d07fd-5f12-4c69-8bb4-a1775a06b6d6 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:29.805923 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9d07fd-5f12-4c69-8bb4-a1775a06b6d6 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:29.863886 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0069148-0180-438b-a2d3-5c803c114ec9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:29.864488 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0069148-0180-438b-a2d3-5c803c114ec9 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:29.864768 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0069148-0180-438b-a2d3-5c803c114ec9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:29.865040 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0069148-0180-438b-a2d3-5c803c114ec9 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:29.865665 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0069148-0180-438b-a2d3-5c803c114ec9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:29.867592 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0069148-0180-438b-a2d3-5c803c114ec9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:29.967409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1411/2813] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:57:29 2022] POST /identity/v3/users => generated 292 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:57:29.971610 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517b10dd-5e73-4f6b-8988-111b90da9380 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:29.982007 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517b10dd-5e73-4f6b-8988-111b90da9380 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:29.986665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778d007e-e364-4988-939f-11ab59fb6a31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:29.987297 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778d007e-e364-4988-939f-11ab59fb6a31 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:29.987593 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778d007e-e364-4988-939f-11ab59fb6a31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:29.987889 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778d007e-e364-4988-939f-11ab59fb6a31 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:29.988611 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-778d007e-e364-4988-939f-11ab59fb6a31 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:29.990372 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-778d007e-e364-4988-939f-11ab59fb6a31 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:30.000335 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1403/2814] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:29 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:57:30.005797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0069148-0180-438b-a2d3-5c803c114ec9 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:30.017333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0069148-0180-438b-a2d3-5c803c114ec9 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:30.022512 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1832665d-4727-4806-92e5-1ffa3ee24157 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:30.023195 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1832665d-4727-4806-92e5-1ffa3ee24157 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:30.023474 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1832665d-4727-4806-92e5-1ffa3ee24157 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:30.023748 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1832665d-4727-4806-92e5-1ffa3ee24157 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:30.024374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1832665d-4727-4806-92e5-1ffa3ee24157 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:30.025948 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1832665d-4727-4806-92e5-1ffa3ee24157 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:30.037791 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1412/2815] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:30 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:57:30.045863 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778d007e-e364-4988-939f-11ab59fb6a31 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:30.052404 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778d007e-e364-4988-939f-11ab59fb6a31 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:30.057050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e7ba12-821a-4de8-aba8-9d95c25ba827 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:30.057813 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e7ba12-821a-4de8-aba8-9d95c25ba827 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:30.058090 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e7ba12-821a-4de8-aba8-9d95c25ba827 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:30.058362 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e7ba12-821a-4de8-aba8-9d95c25ba827 admin admin] PATH_INFO: `/v3/projects/4d19e1665cee485784e9a65afda20547/users/766fe2e340e74cc6aaee25f3555bd3c5/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:30.073882 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e7ba12-821a-4de8-aba8-9d95c25ba827 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4d19e1665cee485784e9a65afda20547, user_id=766fe2e340e74cc6aaee25f3555bd3c5, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:30.076681 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e7ba12-821a-4de8-aba8-9d95c25ba827 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:30.161480 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1404/2816] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:57:30 2022] PUT /identity/v3/projects/4d19e1665cee485784e9a65afda20547/users/766fe2e340e74cc6aaee25f3555bd3c5/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:30.171749 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2ab1a6-2a83-47e9-a3bc-a118d3447239 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:30.172077 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2ab1a6-2a83-47e9-a3bc-a118d3447239 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:30.172403 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2ab1a6-2a83-47e9-a3bc-a118d3447239 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:30.200820 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-cb2ab1a6-2a83-47e9-a3bc-a118d3447239 None None] MFA Rules not processed for user `766fe2e340e74cc6aaee25f3555bd3c5`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:30.202125 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cb2ab1a6-2a83-47e9-a3bc-a118d3447239 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:30.266132 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1413/2817] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:30 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:30.270608 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e7ba12-821a-4de8-aba8-9d95c25ba827 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:30.333003 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e7ba12-821a-4de8-aba8-9d95c25ba827 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:30.337746 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d52a55a-7f7c-449d-916c-0a19e7357be8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:30.338351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d52a55a-7f7c-449d-916c-0a19e7357be8 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:30.338635 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d52a55a-7f7c-449d-916c-0a19e7357be8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:30.338925 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d52a55a-7f7c-449d-916c-0a19e7357be8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:30.339609 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d52a55a-7f7c-449d-916c-0a19e7357be8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:30.341504 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d52a55a-7f7c-449d-916c-0a19e7357be8 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:30.355983 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:30.420811 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1405/2818] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:57:30 2022] POST /identity/v3/projects => generated 385 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:57:30.430600 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2ab1a6-2a83-47e9-a3bc-a118d3447239 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:30.441659 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2ab1a6-2a83-47e9-a3bc-a118d3447239 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:30.451298 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:30.505142 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cbfd70-b602-495a-a33c-fb6a345f1fbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:30.505722 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cbfd70-b602-495a-a33c-fb6a345f1fbf admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:30.506059 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cbfd70-b602-495a-a33c-fb6a345f1fbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:30.506165 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cbfd70-b602-495a-a33c-fb6a345f1fbf admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:30.506767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18cbfd70-b602-495a-a33c-fb6a345f1fbf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:30.508705 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18cbfd70-b602-495a-a33c-fb6a345f1fbf admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:30.606586 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1414/2819] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:57:30 2022] POST /identity/v3/users => generated 293 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:57:30.607783 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d52a55a-7f7c-449d-916c-0a19e7357be8 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:30.619185 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d52a55a-7f7c-449d-916c-0a19e7357be8 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:30.623985 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32fa9f1-5b95-4e1b-80ed-39c55323bad8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:30.624591 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32fa9f1-5b95-4e1b-80ed-39c55323bad8 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:30.624876 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32fa9f1-5b95-4e1b-80ed-39c55323bad8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:30.625145 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32fa9f1-5b95-4e1b-80ed-39c55323bad8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:30.625869 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32fa9f1-5b95-4e1b-80ed-39c55323bad8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:30.627510 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32fa9f1-5b95-4e1b-80ed-39c55323bad8 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:30.640053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1406/2820] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:30 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:57:30.644157 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cbfd70-b602-495a-a33c-fb6a345f1fbf admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:30.654564 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cbfd70-b602-495a-a33c-fb6a345f1fbf admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:30.659368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b77a798-29a1-4279-8253-a451f8905591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:30.660117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b77a798-29a1-4279-8253-a451f8905591 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:30.660400 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b77a798-29a1-4279-8253-a451f8905591 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:30.660675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b77a798-29a1-4279-8253-a451f8905591 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:30.661310 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b77a798-29a1-4279-8253-a451f8905591 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:30.662899 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b77a798-29a1-4279-8253-a451f8905591 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:30.675984 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1415/2821] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:30 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:57:30.677471 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32fa9f1-5b95-4e1b-80ed-39c55323bad8 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:30.687688 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32fa9f1-5b95-4e1b-80ed-39c55323bad8 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:30.692952 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab15183-ec69-4e34-a581-c0ed3efe2ac9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:30.693607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab15183-ec69-4e34-a581-c0ed3efe2ac9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:30.693887 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab15183-ec69-4e34-a581-c0ed3efe2ac9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:30.694159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab15183-ec69-4e34-a581-c0ed3efe2ac9 admin admin] PATH_INFO: `/v3/projects/e19c85ef667b44f7add13a64310e1a16/users/f6c1667263cf4914bb5e2dcdd49f16c2/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:30.710812 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab15183-ec69-4e34-a581-c0ed3efe2ac9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e19c85ef667b44f7add13a64310e1a16, user_id=f6c1667263cf4914bb5e2dcdd49f16c2, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:30.713481 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab15183-ec69-4e34-a581-c0ed3efe2ac9 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:30.853626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1407/2822] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:57:30 2022] PUT /identity/v3/projects/e19c85ef667b44f7add13a64310e1a16/users/f6c1667263cf4914bb5e2dcdd49f16c2/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:30.865416 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7526ed-5e00-4b37-944e-a79043bef676 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:30.865707 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7526ed-5e00-4b37-944e-a79043bef676 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:30.865981 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7526ed-5e00-4b37-944e-a79043bef676 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:30.893643 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9f7526ed-5e00-4b37-944e-a79043bef676 None None] MFA Rules not processed for user `f6c1667263cf4914bb5e2dcdd49f16c2`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:30.895542 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9f7526ed-5e00-4b37-944e-a79043bef676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:30.959557 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1416/2823] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:30 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:30.964106 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab15183-ec69-4e34-a581-c0ed3efe2ac9 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:31.025571 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab15183-ec69-4e34-a581-c0ed3efe2ac9 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:31.030158 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1c8a5b-9344-4ea7-bc84-040600cd45d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:31.030750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1c8a5b-9344-4ea7-bc84-040600cd45d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:31.031063 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1c8a5b-9344-4ea7-bc84-040600cd45d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:31.031333 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1c8a5b-9344-4ea7-bc84-040600cd45d5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:31.031963 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1c8a5b-9344-4ea7-bc84-040600cd45d5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:31.033836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1c8a5b-9344-4ea7-bc84-040600cd45d5 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:31.122535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1408/2824] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:57:30 2022] POST /identity/v3/projects => generated 385 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:57:31.126839 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7526ed-5e00-4b37-944e-a79043bef676 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:31.138758 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7526ed-5e00-4b37-944e-a79043bef676 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:31.197297 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66eac60c-6670-47cd-991b-84043d784f8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:31.197906 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eac60c-6670-47cd-991b-84043d784f8c admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:31.198185 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eac60c-6670-47cd-991b-84043d784f8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:31.198462 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eac60c-6670-47cd-991b-84043d784f8c admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:31.199195 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66eac60c-6670-47cd-991b-84043d784f8c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:31.201093 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66eac60c-6670-47cd-991b-84043d784f8c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:31.293461 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1417/2825] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:57:31 2022] POST /identity/v3/users => generated 299 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:57:31.304039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1c8a5b-9344-4ea7-bc84-040600cd45d5 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:31.314723 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1c8a5b-9344-4ea7-bc84-040600cd45d5 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:31.320102 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b9c060-2661-4a52-9a7c-55812da434a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:31.320707 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b9c060-2661-4a52-9a7c-55812da434a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:31.320989 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b9c060-2661-4a52-9a7c-55812da434a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:31.321265 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b9c060-2661-4a52-9a7c-55812da434a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:31.321913 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b9c060-2661-4a52-9a7c-55812da434a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:31.323533 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b9c060-2661-4a52-9a7c-55812da434a1 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:31.334330 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1409/2826] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:31 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:57:31.338576 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66eac60c-6670-47cd-991b-84043d784f8c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:31.349320 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66eac60c-6670-47cd-991b-84043d784f8c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:31.354128 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ce2cc5-6cd1-461c-9a2f-f36dd63e4eb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:31.354818 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ce2cc5-6cd1-461c-9a2f-f36dd63e4eb8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:31.355094 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ce2cc5-6cd1-461c-9a2f-f36dd63e4eb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:31.355369 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ce2cc5-6cd1-461c-9a2f-f36dd63e4eb8 admin admin] PATH_INFO: `/v3/projects/877799a7bc1940b983b58a4ec5a21495/users/95b0557308b841069d88e7aa0d5b4b36/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:31.364784 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:31.376223 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60ce2cc5-6cd1-461c-9a2f-f36dd63e4eb8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=877799a7bc1940b983b58a4ec5a21495, user_id=95b0557308b841069d88e7aa0d5b4b36, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:31.376223 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60ce2cc5-6cd1-461c-9a2f-f36dd63e4eb8 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:31.463182 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1418/2827] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:57:31 2022] PUT /identity/v3/projects/877799a7bc1940b983b58a4ec5a21495/users/95b0557308b841069d88e7aa0d5b4b36/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:31.467392 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b9c060-2661-4a52-9a7c-55812da434a1 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:31.478119 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b9c060-2661-4a52-9a7c-55812da434a1 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:31.534225 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648cf463-a1c9-4ae3-a8fd-192c299aa4e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:31.535471 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648cf463-a1c9-4ae3-a8fd-192c299aa4e0 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:31.535755 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648cf463-a1c9-4ae3-a8fd-192c299aa4e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:31.536030 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648cf463-a1c9-4ae3-a8fd-192c299aa4e0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:31.536679 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648cf463-a1c9-4ae3-a8fd-192c299aa4e0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:31.538511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648cf463-a1c9-4ae3-a8fd-192c299aa4e0 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:31.548588 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1410/2828] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:31 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:57:31.553218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ce2cc5-6cd1-461c-9a2f-f36dd63e4eb8 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:31.564059 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ce2cc5-6cd1-461c-9a2f-f36dd63e4eb8 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:31.569184 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfcd1fa-2056-48c2-8f11-62444cd09b66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:31.580828 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfcd1fa-2056-48c2-8f11-62444cd09b66 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:31.580828 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfcd1fa-2056-48c2-8f11-62444cd09b66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:31.580828 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfcd1fa-2056-48c2-8f11-62444cd09b66 admin admin] PATH_INFO: `/v3/projects/877799a7bc1940b983b58a4ec5a21495/users/95b0557308b841069d88e7aa0d5b4b36/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:31.580828 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfcd1fa-2056-48c2-8f11-62444cd09b66 admin admin] RBAC: Authorizing `identity:create_grant(project_id=877799a7bc1940b983b58a4ec5a21495, user_id=95b0557308b841069d88e7aa0d5b4b36, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:31.580828 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfcd1fa-2056-48c2-8f11-62444cd09b66 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:31.585618 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1419/2829] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:57:31 2022] PUT /identity/v3/projects/877799a7bc1940b983b58a4ec5a21495/users/95b0557308b841069d88e7aa0d5b4b36/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:57:31.595752 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf33dca0-e798-4384-9494-c1b01e35ee57 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:31.596046 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf33dca0-e798-4384-9494-c1b01e35ee57 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:31.596326 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf33dca0-e798-4384-9494-c1b01e35ee57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:31.625626 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-bf33dca0-e798-4384-9494-c1b01e35ee57 None None] MFA Rules not processed for user `95b0557308b841069d88e7aa0d5b4b36`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:31.626818 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-bf33dca0-e798-4384-9494-c1b01e35ee57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:31.645162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:31.697426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1411/2830] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:31 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:31.709756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfcd1fa-2056-48c2-8f11-62444cd09b66 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:31.711602 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4bfcd1fa-2056-48c2-8f11-62444cd09b66 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:31.776021 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfcd1fa-2056-48c2-8f11-62444cd09b66 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:31.779928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8fb8d01-6503-4eca-a3e9-836a5ecc7ee9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:31.780551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fb8d01-6503-4eca-a3e9-836a5ecc7ee9 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:31.780833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fb8d01-6503-4eca-a3e9-836a5ecc7ee9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:31.781111 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fb8d01-6503-4eca-a3e9-836a5ecc7ee9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:31.841124 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8fb8d01-6503-4eca-a3e9-836a5ecc7ee9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:31.842574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8fb8d01-6503-4eca-a3e9-836a5ecc7ee9 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:31.851645 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1420/2831] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:31 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:57:33.947999 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf33dca0-e798-4384-9494-c1b01e35ee57 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:33.957808 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf33dca0-e798-4384-9494-c1b01e35ee57 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:33.961965 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646555f2-0a02-477d-94ae-bc6388030b98 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:33.962553 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646555f2-0a02-477d-94ae-bc6388030b98 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:33.962841 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646555f2-0a02-477d-94ae-bc6388030b98 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:33.963130 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646555f2-0a02-477d-94ae-bc6388030b98 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:34.018029 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646555f2-0a02-477d-94ae-bc6388030b98 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:34.019515 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646555f2-0a02-477d-94ae-bc6388030b98 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:34.028829 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1412/2832] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:33 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:57:35.498913 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8fb8d01-6503-4eca-a3e9-836a5ecc7ee9 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:35.500442 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f8fb8d01-6503-4eca-a3e9-836a5ecc7ee9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:35.511713 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8fb8d01-6503-4eca-a3e9-836a5ecc7ee9 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:35.515348 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ce490a-5c11-48a2-9b7b-5b991617d5ed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:35.515943 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ce490a-5c11-48a2-9b7b-5b991617d5ed service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:35.516226 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ce490a-5c11-48a2-9b7b-5b991617d5ed service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:35.516508 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ce490a-5c11-48a2-9b7b-5b991617d5ed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:35.526068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60ce490a-5c11-48a2-9b7b-5b991617d5ed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:35.527536 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60ce490a-5c11-48a2-9b7b-5b991617d5ed service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:35.538193 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1421/2833] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:35 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:57:40.151736 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646555f2-0a02-477d-94ae-bc6388030b98 service cinder] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:40.233709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646555f2-0a02-477d-94ae-bc6388030b98 service cinder] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:40.241790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1b361f-0df1-4b64-864f-bc98e36829f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:40.242445 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1b361f-0df1-4b64-864f-bc98e36829f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:40.242728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1b361f-0df1-4b64-864f-bc98e36829f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:40.243042 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1b361f-0df1-4b64-864f-bc98e36829f7 admin admin] PATH_INFO: `/v3/users/766fe2e340e74cc6aaee25f3555bd3c5` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:40.244290 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1b361f-0df1-4b64-864f-bc98e36829f7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=766fe2e340e74cc6aaee25f3555bd3c5)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:40.246219 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1b361f-0df1-4b64-864f-bc98e36829f7 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:40.434605 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-ca1b361f-0df1-4b64-864f-bc98e36829f7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '766fe2e340e74cc6aaee25f3555bd3c5'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:40.498422 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-ca1b361f-0df1-4b64-864f-bc98e36829f7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '766fe2e340e74cc6aaee25f3555bd3c5'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:40.500135 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:40.520942 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-ca1b361f-0df1-4b64-864f-bc98e36829f7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '766fe2e340e74cc6aaee25f3555bd3c5'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:40.521758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-ca1b361f-0df1-4b64-864f-bc98e36829f7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '766fe2e340e74cc6aaee25f3555bd3c5'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:40.529191 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-ca1b361f-0df1-4b64-864f-bc98e36829f7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '766fe2e340e74cc6aaee25f3555bd3c5'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:40.535625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1413/2834] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:57:40 2022] DELETE /identity/v3/users/766fe2e340e74cc6aaee25f3555bd3c5 => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:40.543913 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316e021c-60d2-420f-8537-bee214efac72 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:40.544503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316e021c-60d2-420f-8537-bee214efac72 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:40.544503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316e021c-60d2-420f-8537-bee214efac72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:40.572028 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-316e021c-60d2-420f-8537-bee214efac72 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:40.573284 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-316e021c-60d2-420f-8537-bee214efac72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:40.652381 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1422/2835] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:40.665873 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1b361f-0df1-4b64-864f-bc98e36829f7 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:40.667410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-ca1b361f-0df1-4b64-864f-bc98e36829f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:40.739709 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1b361f-0df1-4b64-864f-bc98e36829f7 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:40.745897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f09a607-18aa-4115-96fc-f325165ceb12 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:40.746867 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f09a607-18aa-4115-96fc-f325165ceb12 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:40.747352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f09a607-18aa-4115-96fc-f325165ceb12 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:40.747799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f09a607-18aa-4115-96fc-f325165ceb12 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:40.759604 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f09a607-18aa-4115-96fc-f325165ceb12 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:40.761250 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f09a607-18aa-4115-96fc-f325165ceb12 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:40.771254 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1414/2836] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:57:41.168138 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316e021c-60d2-420f-8537-bee214efac72 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:41.169624 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-316e021c-60d2-420f-8537-bee214efac72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:41.182678 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316e021c-60d2-420f-8537-bee214efac72 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:41.187929 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb5a8ea-f46f-41dd-9e2b-f35f706503fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:41.188422 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb5a8ea-f46f-41dd-9e2b-f35f706503fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:41.188706 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb5a8ea-f46f-41dd-9e2b-f35f706503fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:41.188982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb5a8ea-f46f-41dd-9e2b-f35f706503fe admin admin] PATH_INFO: `/v3/projects/4d19e1665cee485784e9a65afda20547` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:41.190137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb5a8ea-f46f-41dd-9e2b-f35f706503fe admin admin] RBAC: Authorizing `identity:delete_project(project_id=4d19e1665cee485784e9a65afda20547)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:41.192087 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb5a8ea-f46f-41dd-9e2b-f35f706503fe admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:41.203616 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:41.290574 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9fb5a8ea-f46f-41dd-9e2b-f35f706503fe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4d19e1665cee485784e9a65afda20547', 'request_id': 'req-9fb5a8ea-f46f-41dd-9e2b-f35f706503fe'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:41.296889 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9fb5a8ea-f46f-41dd-9e2b-f35f706503fe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4d19e1665cee485784e9a65afda20547', 'request_id': 'req-9fb5a8ea-f46f-41dd-9e2b-f35f706503fe'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:41.382623 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9fb5a8ea-f46f-41dd-9e2b-f35f706503fe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4d19e1665cee485784e9a65afda20547', 'request_id': 'req-9fb5a8ea-f46f-41dd-9e2b-f35f706503fe'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:41.394735 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9fb5a8ea-f46f-41dd-9e2b-f35f706503fe admin admin] The token cache is being invalidate because project 4d19e1665cee485784e9a65afda20547 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:57:41.395092 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9fb5a8ea-f46f-41dd-9e2b-f35f706503fe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:41.395721 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9fb5a8ea-f46f-41dd-9e2b-f35f706503fe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:41.398191 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1423/2837] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:57:41 2022] DELETE /identity/v3/projects/4d19e1665cee485784e9a65afda20547 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:41.402334 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f09a607-18aa-4115-96fc-f325165ceb12 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:41.413576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f09a607-18aa-4115-96fc-f325165ceb12 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:41.469564 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a98404b-b491-4db8-8ba0-851a62a2324b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:41.470190 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a98404b-b491-4db8-8ba0-851a62a2324b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:41.470476 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a98404b-b491-4db8-8ba0-851a62a2324b admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:41.470750 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a98404b-b491-4db8-8ba0-851a62a2324b admin admin] PATH_INFO: `/v3/users/f6c1667263cf4914bb5e2dcdd49f16c2` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:41.471909 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a98404b-b491-4db8-8ba0-851a62a2324b admin admin] RBAC: Authorizing `identity:delete_user(user_id=f6c1667263cf4914bb5e2dcdd49f16c2)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:41.473823 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a98404b-b491-4db8-8ba0-851a62a2324b admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:41.654320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1a98404b-b491-4db8-8ba0-851a62a2324b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f6c1667263cf4914bb5e2dcdd49f16c2'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:41.738858 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1a98404b-b491-4db8-8ba0-851a62a2324b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f6c1667263cf4914bb5e2dcdd49f16c2'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:41.751868 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1a98404b-b491-4db8-8ba0-851a62a2324b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f6c1667263cf4914bb5e2dcdd49f16c2'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:41.752549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1a98404b-b491-4db8-8ba0-851a62a2324b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f6c1667263cf4914bb5e2dcdd49f16c2'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:41.757962 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:41.760953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-1a98404b-b491-4db8-8ba0-851a62a2324b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f6c1667263cf4914bb5e2dcdd49f16c2'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:41.767103 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1415/2838] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:57:41 2022] DELETE /identity/v3/users/f6c1667263cf4914bb5e2dcdd49f16c2 => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:42.186197 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb5a8ea-f46f-41dd-9e2b-f35f706503fe admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:42.186197 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9fb5a8ea-f46f-41dd-9e2b-f35f706503fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:42.196120 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb5a8ea-f46f-41dd-9e2b-f35f706503fe admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:42.253895 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c760537-ad0f-454f-b044-7d5f63c5b225 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:42.254473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c760537-ad0f-454f-b044-7d5f63c5b225 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:42.254737 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c760537-ad0f-454f-b044-7d5f63c5b225 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:42.255044 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c760537-ad0f-454f-b044-7d5f63c5b225 admin admin] PATH_INFO: `/v3/projects/e19c85ef667b44f7add13a64310e1a16` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:42.256166 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c760537-ad0f-454f-b044-7d5f63c5b225 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e19c85ef667b44f7add13a64310e1a16)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:42.258056 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c760537-ad0f-454f-b044-7d5f63c5b225 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:42.379857 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5c760537-ad0f-454f-b044-7d5f63c5b225 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e19c85ef667b44f7add13a64310e1a16', 'request_id': 'req-5c760537-ad0f-454f-b044-7d5f63c5b225'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:42.386222 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5c760537-ad0f-454f-b044-7d5f63c5b225 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e19c85ef667b44f7add13a64310e1a16', 'request_id': 'req-5c760537-ad0f-454f-b044-7d5f63c5b225'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:42.458665 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5c760537-ad0f-454f-b044-7d5f63c5b225 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e19c85ef667b44f7add13a64310e1a16', 'request_id': 'req-5c760537-ad0f-454f-b044-7d5f63c5b225'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:42.470431 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5c760537-ad0f-454f-b044-7d5f63c5b225 admin admin] The token cache is being invalidate because project e19c85ef667b44f7add13a64310e1a16 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:57:42.470719 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5c760537-ad0f-454f-b044-7d5f63c5b225 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:42.471374 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5c760537-ad0f-454f-b044-7d5f63c5b225 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:42.472851 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1424/2839] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:57:42 2022] DELETE /identity/v3/projects/e19c85ef667b44f7add13a64310e1a16 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:42.477581 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a98404b-b491-4db8-8ba0-851a62a2324b admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:42.479111 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1a98404b-b491-4db8-8ba0-851a62a2324b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:42.491518 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a98404b-b491-4db8-8ba0-851a62a2324b admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:42.550459 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5079d166-45c5-44c1-a688-56c4c6c8a31b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:42.551115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5079d166-45c5-44c1-a688-56c4c6c8a31b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:42.551396 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5079d166-45c5-44c1-a688-56c4c6c8a31b admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:42.551675 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5079d166-45c5-44c1-a688-56c4c6c8a31b admin admin] PATH_INFO: `/v3/users/95b0557308b841069d88e7aa0d5b4b36` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:42.552883 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5079d166-45c5-44c1-a688-56c4c6c8a31b admin admin] RBAC: Authorizing `identity:delete_user(user_id=95b0557308b841069d88e7aa0d5b4b36)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:42.554790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5079d166-45c5-44c1-a688-56c4c6c8a31b admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:42.779676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5079d166-45c5-44c1-a688-56c4c6c8a31b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '95b0557308b841069d88e7aa0d5b4b36'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:42.876314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5079d166-45c5-44c1-a688-56c4c6c8a31b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '95b0557308b841069d88e7aa0d5b4b36'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:42.889684 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5079d166-45c5-44c1-a688-56c4c6c8a31b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '95b0557308b841069d88e7aa0d5b4b36'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:42.890367 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5079d166-45c5-44c1-a688-56c4c6c8a31b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '95b0557308b841069d88e7aa0d5b4b36'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:42.898101 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-5079d166-45c5-44c1-a688-56c4c6c8a31b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '95b0557308b841069d88e7aa0d5b4b36'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:42.904376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1416/2840] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:57:42 2022] DELETE /identity/v3/users/95b0557308b841069d88e7aa0d5b4b36 => generated 0 bytes in 427 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:43.353452 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c760537-ad0f-454f-b044-7d5f63c5b225 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:43.355295 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5c760537-ad0f-454f-b044-7d5f63c5b225 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:43.378459 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c760537-ad0f-454f-b044-7d5f63c5b225 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:43.407061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:43.446856 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a5feae-17fb-4a89-a628-438872dd7eeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:43.456436 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a5feae-17fb-4a89-a628-438872dd7eeb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:43.456436 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a5feae-17fb-4a89-a628-438872dd7eeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:43.456436 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a5feae-17fb-4a89-a628-438872dd7eeb admin admin] PATH_INFO: `/v3/projects/877799a7bc1940b983b58a4ec5a21495` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:43.456436 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a5feae-17fb-4a89-a628-438872dd7eeb admin admin] RBAC: Authorizing `identity:delete_project(project_id=877799a7bc1940b983b58a4ec5a21495)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:43.456436 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a5feae-17fb-4a89-a628-438872dd7eeb admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:43.584933 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-89a5feae-17fb-4a89-a628-438872dd7eeb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '877799a7bc1940b983b58a4ec5a21495', 'request_id': 'req-89a5feae-17fb-4a89-a628-438872dd7eeb'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:43.591338 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-89a5feae-17fb-4a89-a628-438872dd7eeb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '877799a7bc1940b983b58a4ec5a21495', 'request_id': 'req-89a5feae-17fb-4a89-a628-438872dd7eeb'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:43.699307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-89a5feae-17fb-4a89-a628-438872dd7eeb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '877799a7bc1940b983b58a4ec5a21495', 'request_id': 'req-89a5feae-17fb-4a89-a628-438872dd7eeb'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:43.710477 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-89a5feae-17fb-4a89-a628-438872dd7eeb admin admin] The token cache is being invalidate because project 877799a7bc1940b983b58a4ec5a21495 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:57:43.710854 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-89a5feae-17fb-4a89-a628-438872dd7eeb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:43.711938 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-89a5feae-17fb-4a89-a628-438872dd7eeb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:43.713540 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1425/2841] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:57:43 2022] DELETE /identity/v3/projects/877799a7bc1940b983b58a4ec5a21495 => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:43.718734 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df54d6b-79c2-403a-a282-f4adb0044746 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:43.719258 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df54d6b-79c2-403a-a282-f4adb0044746 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:43.719379 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df54d6b-79c2-403a-a282-f4adb0044746 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:43.739512 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-3df54d6b-79c2-403a-a282-f4adb0044746 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:43.741858 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3df54d6b-79c2-403a-a282-f4adb0044746 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:43.801435 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1417/2842] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:43.811590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765f3a07-45d5-46e7-a6ed-2914240e44c4 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:43.812032 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765f3a07-45d5-46e7-a6ed-2914240e44c4 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:43.812317 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765f3a07-45d5-46e7-a6ed-2914240e44c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:43.836329 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-765f3a07-45d5-46e7-a6ed-2914240e44c4 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:43.837475 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-765f3a07-45d5-46e7-a6ed-2914240e44c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:43.845595 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1426/2843] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:43.852856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df54d6b-79c2-403a-a282-f4adb0044746 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:43.863428 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df54d6b-79c2-403a-a282-f4adb0044746 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:43.872987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234e0a81-6eef-4b3d-bc10-4b4c67372f7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:43.872987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234e0a81-6eef-4b3d-bc10-4b4c67372f7f admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:43.872987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234e0a81-6eef-4b3d-bc10-4b4c67372f7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:43.872987 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234e0a81-6eef-4b3d-bc10-4b4c67372f7f admin admin] PATH_INFO: `/v3/domains` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:43.873346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234e0a81-6eef-4b3d-bc10-4b4c67372f7f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81761) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:57:43.873346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234e0a81-6eef-4b3d-bc10-4b4c67372f7f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:43.873346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234e0a81-6eef-4b3d-bc10-4b4c67372f7f admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:43.883268 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:43.886834 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1418/2844] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:57:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:57:43.891130 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765f3a07-45d5-46e7-a6ed-2914240e44c4 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:43.901023 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765f3a07-45d5-46e7-a6ed-2914240e44c4 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:43.906586 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65be437-8bf6-4c75-8fba-baeb16617ab1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:43.907222 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65be437-8bf6-4c75-8fba-baeb16617ab1 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:43.907503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65be437-8bf6-4c75-8fba-baeb16617ab1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:43.907776 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65be437-8bf6-4c75-8fba-baeb16617ab1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:43.908418 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65be437-8bf6-4c75-8fba-baeb16617ab1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:43.910358 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65be437-8bf6-4c75-8fba-baeb16617ab1 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:44.037576 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1427/2845] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:57:43 2022] POST /identity/v3/projects => generated 371 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:57:44.042103 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234e0a81-6eef-4b3d-bc10-4b4c67372f7f admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:44.052124 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234e0a81-6eef-4b3d-bc10-4b4c67372f7f admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:44.110646 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611b560e-6799-47ea-8b63-0b51d22a2178 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:44.111261 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611b560e-6799-47ea-8b63-0b51d22a2178 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:44.111530 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611b560e-6799-47ea-8b63-0b51d22a2178 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:44.111799 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611b560e-6799-47ea-8b63-0b51d22a2178 admin admin] PATH_INFO: `/v3/users` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:44.112444 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611b560e-6799-47ea-8b63-0b51d22a2178 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:44.114439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611b560e-6799-47ea-8b63-0b51d22a2178 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:44.195947 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1419/2846] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:57:44 2022] POST /identity/v3/users => generated 286 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:57:44.201503 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65be437-8bf6-4c75-8fba-baeb16617ab1 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:44.212278 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65be437-8bf6-4c75-8fba-baeb16617ab1 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:44.217616 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce77193-c62f-4661-ae1a-b738176fc18a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:44.220470 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce77193-c62f-4661-ae1a-b738176fc18a admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:44.220470 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce77193-c62f-4661-ae1a-b738176fc18a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:44.220470 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce77193-c62f-4661-ae1a-b738176fc18a admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:44.220470 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce77193-c62f-4661-ae1a-b738176fc18a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:44.221554 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce77193-c62f-4661-ae1a-b738176fc18a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:44.233091 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1428/2847] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:44 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:57:44.237031 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611b560e-6799-47ea-8b63-0b51d22a2178 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:44.247747 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611b560e-6799-47ea-8b63-0b51d22a2178 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:44.252426 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014e26cf-e2b1-4890-8751-f26b4edcc1dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:44.253025 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014e26cf-e2b1-4890-8751-f26b4edcc1dd admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:44.253302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014e26cf-e2b1-4890-8751-f26b4edcc1dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:44.253574 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014e26cf-e2b1-4890-8751-f26b4edcc1dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:44.254213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014e26cf-e2b1-4890-8751-f26b4edcc1dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:44.255957 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014e26cf-e2b1-4890-8751-f26b4edcc1dd admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:44.271620 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1420/2848] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:44 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:57:44.277896 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce77193-c62f-4661-ae1a-b738176fc18a admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:44.289377 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce77193-c62f-4661-ae1a-b738176fc18a admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:44.296887 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8727d148-cb53-4d17-a1e2-251452175903 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:44.297797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8727d148-cb53-4d17-a1e2-251452175903 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:44.298084 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8727d148-cb53-4d17-a1e2-251452175903 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:44.298392 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8727d148-cb53-4d17-a1e2-251452175903 admin admin] PATH_INFO: `/v3/projects/ada320ab9e1644f29fd613c173ffe0a8/users/6794c30b34984af2b3eb6c0902a6157d/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:44.315306 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8727d148-cb53-4d17-a1e2-251452175903 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ada320ab9e1644f29fd613c173ffe0a8, user_id=6794c30b34984af2b3eb6c0902a6157d, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:44.318001 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8727d148-cb53-4d17-a1e2-251452175903 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:44.391518 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1429/2849] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:57:44 2022] PUT /identity/v3/projects/ada320ab9e1644f29fd613c173ffe0a8/users/6794c30b34984af2b3eb6c0902a6157d/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:44.397134 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59125276-01a4-44f3-9a6b-13c918c125b2 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:44.397428 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59125276-01a4-44f3-9a6b-13c918c125b2 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:44.397706 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59125276-01a4-44f3-9a6b-13c918c125b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:44.417897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-59125276-01a4-44f3-9a6b-13c918c125b2 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:44.419127 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-59125276-01a4-44f3-9a6b-13c918c125b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:44.480108 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1421/2850] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:44.491047 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8727d148-cb53-4d17-a1e2-251452175903 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:44.493218 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8727d148-cb53-4d17-a1e2-251452175903 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:44.551052 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8727d148-cb53-4d17-a1e2-251452175903 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:44.554747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f69b4e5-1e30-4346-af04-173e46e0f415 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:44.555383 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f69b4e5-1e30-4346-af04-173e46e0f415 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:44.555660 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f69b4e5-1e30-4346-af04-173e46e0f415 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:44.555930 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f69b4e5-1e30-4346-af04-173e46e0f415 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:44.564937 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f69b4e5-1e30-4346-af04-173e46e0f415 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:44.566361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f69b4e5-1e30-4346-af04-173e46e0f415 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:44.570603 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:44.576963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1430/2851] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:57:46.024585 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb3deb0-e8bc-4da9-bf4e-ddad5bebdd40 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:46.025248 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb3deb0-e8bc-4da9-bf4e-ddad5bebdd40 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:46.025248 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb3deb0-e8bc-4da9-bf4e-ddad5bebdd40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:46.058881 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-8eb3deb0-e8bc-4da9-bf4e-ddad5bebdd40 None None] MFA Rules not processed for user `6794c30b34984af2b3eb6c0902a6157d`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:46.060499 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-8eb3deb0-e8bc-4da9-bf4e-ddad5bebdd40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:46.131370 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1422/2852] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:46 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:46.143590 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f69b4e5-1e30-4346-af04-173e46e0f415 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:46.145073 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7f69b4e5-1e30-4346-af04-173e46e0f415 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:46.205282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f69b4e5-1e30-4346-af04-173e46e0f415 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:46.210325 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bca9797-0dad-4246-9837-8fb4d1c39c15 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:46.211679 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bca9797-0dad-4246-9837-8fb4d1c39c15 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:46.211973 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bca9797-0dad-4246-9837-8fb4d1c39c15 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:46.212250 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bca9797-0dad-4246-9837-8fb4d1c39c15 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:46.220689 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bca9797-0dad-4246-9837-8fb4d1c39c15 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:46.222792 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bca9797-0dad-4246-9837-8fb4d1c39c15 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:46.233364 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1431/2853] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:46 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:57:47.985852 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a477d9a-bb65-44e9-85fe-3293222877b3 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:47.986475 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a477d9a-bb65-44e9-85fe-3293222877b3 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:47.986581 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a477d9a-bb65-44e9-85fe-3293222877b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:48.006769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-1a477d9a-bb65-44e9-85fe-3293222877b3 None None] MFA Rules not processed for user `6794c30b34984af2b3eb6c0902a6157d`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:48.008004 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-1a477d9a-bb65-44e9-85fe-3293222877b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:48.015289 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1423/2854] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:47 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:54.679180 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bca9797-0dad-4246-9837-8fb4d1c39c15 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:54.680793 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1bca9797-0dad-4246-9837-8fb4d1c39c15 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:54.695682 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bca9797-0dad-4246-9837-8fb4d1c39c15 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:54.700443 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b6624b-6b90-4bba-a0cf-b4ba70e03462 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:54.701061 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b6624b-6b90-4bba-a0cf-b4ba70e03462 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:54.701336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b6624b-6b90-4bba-a0cf-b4ba70e03462 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:54.701607 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b6624b-6b90-4bba-a0cf-b4ba70e03462 admin admin] PATH_INFO: `/v3/users/4688254278514dda955efd2b185f37a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:54.702722 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b6624b-6b90-4bba-a0cf-b4ba70e03462 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4688254278514dda955efd2b185f37a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:54.704691 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b6624b-6b90-4bba-a0cf-b4ba70e03462 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:54.906931 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d2b6624b-6b90-4bba-a0cf-b4ba70e03462 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4688254278514dda955efd2b185f37a2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:54.918747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d2b6624b-6b90-4bba-a0cf-b4ba70e03462 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4688254278514dda955efd2b185f37a2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:54.990104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d2b6624b-6b90-4bba-a0cf-b4ba70e03462 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4688254278514dda955efd2b185f37a2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:54.990714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d2b6624b-6b90-4bba-a0cf-b4ba70e03462 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4688254278514dda955efd2b185f37a2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:54.999079 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d2b6624b-6b90-4bba-a0cf-b4ba70e03462 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4688254278514dda955efd2b185f37a2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:55.004995 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1432/2855] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:57:54 2022] DELETE /identity/v3/users/4688254278514dda955efd2b185f37a2 => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:55.010091 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb448b6-cbd0-48bd-855f-788521c6d75e None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:55.010588 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb448b6-cbd0-48bd-855f-788521c6d75e None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:55.010690 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb448b6-cbd0-48bd-855f-788521c6d75e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:55.017655 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:55.030768 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-cfb448b6-cbd0-48bd-855f-788521c6d75e None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:55.031921 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-cfb448b6-cbd0-48bd-855f-788521c6d75e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:55.090070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1424/2856] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:55.099740 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b6624b-6b90-4bba-a0cf-b4ba70e03462 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:55.101123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d2b6624b-6b90-4bba-a0cf-b4ba70e03462 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:55.158063 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b6624b-6b90-4bba-a0cf-b4ba70e03462 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:55.161739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e4a3a7-db94-4db9-992e-67155bc56186 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:55.162304 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e4a3a7-db94-4db9-992e-67155bc56186 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:55.165611 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e4a3a7-db94-4db9-992e-67155bc56186 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:55.165611 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e4a3a7-db94-4db9-992e-67155bc56186 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:55.171691 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e4a3a7-db94-4db9-992e-67155bc56186 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:55.173097 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e4a3a7-db94-4db9-992e-67155bc56186 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:55.182822 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1433/2857] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:57:55.603438 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb448b6-cbd0-48bd-855f-788521c6d75e None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:55.604870 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-cfb448b6-cbd0-48bd-855f-788521c6d75e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:55.619039 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb448b6-cbd0-48bd-855f-788521c6d75e None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:55.623903 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685ffca0-58a0-4196-92fd-293fb6dfcd29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:55.624675 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685ffca0-58a0-4196-92fd-293fb6dfcd29 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:55.624961 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685ffca0-58a0-4196-92fd-293fb6dfcd29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:55.625237 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685ffca0-58a0-4196-92fd-293fb6dfcd29 admin admin] PATH_INFO: `/v3/projects/72b3d7c3abda448bb208fabc93cc2ec0` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:55.626754 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685ffca0-58a0-4196-92fd-293fb6dfcd29 admin admin] RBAC: Authorizing `identity:delete_project(project_id=72b3d7c3abda448bb208fabc93cc2ec0)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:55.628742 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685ffca0-58a0-4196-92fd-293fb6dfcd29 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:55.731618 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-685ffca0-58a0-4196-92fd-293fb6dfcd29 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '72b3d7c3abda448bb208fabc93cc2ec0', 'request_id': 'req-685ffca0-58a0-4196-92fd-293fb6dfcd29'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:55.812500 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-685ffca0-58a0-4196-92fd-293fb6dfcd29 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '72b3d7c3abda448bb208fabc93cc2ec0', 'request_id': 'req-685ffca0-58a0-4196-92fd-293fb6dfcd29'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:55.818166 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-685ffca0-58a0-4196-92fd-293fb6dfcd29 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '72b3d7c3abda448bb208fabc93cc2ec0', 'request_id': 'req-685ffca0-58a0-4196-92fd-293fb6dfcd29'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:55.829339 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-685ffca0-58a0-4196-92fd-293fb6dfcd29 admin admin] The token cache is being invalidate because project 72b3d7c3abda448bb208fabc93cc2ec0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:57:55.829621 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-685ffca0-58a0-4196-92fd-293fb6dfcd29 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:55.830241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-685ffca0-58a0-4196-92fd-293fb6dfcd29 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:57:55.831741 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1425/2858] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:57:55 2022] DELETE /identity/v3/projects/72b3d7c3abda448bb208fabc93cc2ec0 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:57:55.837096 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5e2bcb-3720-4b22-954d-80696ba2515d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:55.837584 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5e2bcb-3720-4b22-954d-80696ba2515d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:55.837684 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5e2bcb-3720-4b22-954d-80696ba2515d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:55.859610 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1e5e2bcb-3720-4b22-954d-80696ba2515d None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:55.860814 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1e5e2bcb-3720-4b22-954d-80696ba2515d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:55.870701 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:55.922865 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1434/2859] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:55.931219 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0aa5ed-860f-4729-8ac4-0430d3abb315 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:55.931492 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0aa5ed-860f-4729-8ac4-0430d3abb315 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:55.932452 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0aa5ed-860f-4729-8ac4-0430d3abb315 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:55.953394 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-8c0aa5ed-860f-4729-8ac4-0430d3abb315 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:55.954590 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-8c0aa5ed-860f-4729-8ac4-0430d3abb315 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:55.963793 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1426/2860] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:55.968331 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5e2bcb-3720-4b22-954d-80696ba2515d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:55.978730 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5e2bcb-3720-4b22-954d-80696ba2515d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:55.986447 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3798002a-69b0-4fd6-af61-b86aae2c630f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:55.987200 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3798002a-69b0-4fd6-af61-b86aae2c630f admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:55.987543 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3798002a-69b0-4fd6-af61-b86aae2c630f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:55.987819 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3798002a-69b0-4fd6-af61-b86aae2c630f admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:55.988464 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3798002a-69b0-4fd6-af61-b86aae2c630f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:57:55.988762 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3798002a-69b0-4fd6-af61-b86aae2c630f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:55.990344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3798002a-69b0-4fd6-af61-b86aae2c630f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:56.002982 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1435/2861] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:57:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:57:56.006886 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0aa5ed-860f-4729-8ac4-0430d3abb315 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:56.016728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0aa5ed-860f-4729-8ac4-0430d3abb315 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:56.021607 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d354ba-8151-4c94-b2d7-90f0fd992d06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:56.022231 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d354ba-8151-4c94-b2d7-90f0fd992d06 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:56.022509 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d354ba-8151-4c94-b2d7-90f0fd992d06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:56.022779 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d354ba-8151-4c94-b2d7-90f0fd992d06 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:56.024037 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d354ba-8151-4c94-b2d7-90f0fd992d06 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:56.025909 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d354ba-8151-4c94-b2d7-90f0fd992d06 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:56.111217 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1427/2862] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:57:56 2022] POST /identity/v3/projects => generated 375 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:57:56.115343 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3798002a-69b0-4fd6-af61-b86aae2c630f admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:56.125368 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3798002a-69b0-4fd6-af61-b86aae2c630f admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:56.181457 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5cad9b0-af9a-4a32-ac2e-f885e7cf5ddb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:56.182053 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cad9b0-af9a-4a32-ac2e-f885e7cf5ddb admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:56.182336 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cad9b0-af9a-4a32-ac2e-f885e7cf5ddb admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:56.182607 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cad9b0-af9a-4a32-ac2e-f885e7cf5ddb admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:56.183343 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5cad9b0-af9a-4a32-ac2e-f885e7cf5ddb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:56.185468 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5cad9b0-af9a-4a32-ac2e-f885e7cf5ddb admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:56.338876 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1436/2863] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:57:56 2022] POST /identity/v3/users => generated 288 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:57:56.342732 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d354ba-8151-4c94-b2d7-90f0fd992d06 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:56.353488 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d354ba-8151-4c94-b2d7-90f0fd992d06 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:56.358053 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f417611-dc81-43ed-b40b-e3355f6724c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:56.358653 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f417611-dc81-43ed-b40b-e3355f6724c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:56.358964 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f417611-dc81-43ed-b40b-e3355f6724c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:56.359246 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f417611-dc81-43ed-b40b-e3355f6724c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:56.360006 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f417611-dc81-43ed-b40b-e3355f6724c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:56.361651 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f417611-dc81-43ed-b40b-e3355f6724c4 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:56.372002 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1428/2864] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:57:56.375882 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5cad9b0-af9a-4a32-ac2e-f885e7cf5ddb admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:56.386716 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5cad9b0-af9a-4a32-ac2e-f885e7cf5ddb admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:56.391505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0660cd56-644e-4c8d-9fdb-8e4b11768b56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:56.392118 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0660cd56-644e-4c8d-9fdb-8e4b11768b56 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:56.392416 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0660cd56-644e-4c8d-9fdb-8e4b11768b56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:56.392694 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0660cd56-644e-4c8d-9fdb-8e4b11768b56 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:56.393329 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0660cd56-644e-4c8d-9fdb-8e4b11768b56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:56.394935 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0660cd56-644e-4c8d-9fdb-8e4b11768b56 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:56.405675 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1437/2865] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:57:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:57:56.409667 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f417611-dc81-43ed-b40b-e3355f6724c4 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:56.419761 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f417611-dc81-43ed-b40b-e3355f6724c4 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:56.424271 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14e4203-4c2a-4a96-9184-a0ec581c7579 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:56.424911 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14e4203-4c2a-4a96-9184-a0ec581c7579 admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:56.425199 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14e4203-4c2a-4a96-9184-a0ec581c7579 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:56.425479 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14e4203-4c2a-4a96-9184-a0ec581c7579 admin admin] PATH_INFO: `/v3/projects/548a2763e5c146c783f15114b4070a0d/users/a28d6bcc3a664d498d07890e2cab7f1a/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:56.440671 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c14e4203-4c2a-4a96-9184-a0ec581c7579 admin admin] RBAC: Authorizing `identity:create_grant(project_id=548a2763e5c146c783f15114b4070a0d, user_id=a28d6bcc3a664d498d07890e2cab7f1a, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:56.443370 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c14e4203-4c2a-4a96-9184-a0ec581c7579 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:56.503884 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1429/2866] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:57:56 2022] PUT /identity/v3/projects/548a2763e5c146c783f15114b4070a0d/users/a28d6bcc3a664d498d07890e2cab7f1a/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:57:56.569857 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63398151-dd22-458a-8089-141353bce9e0 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:56.570736 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63398151-dd22-458a-8089-141353bce9e0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:56.571052 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63398151-dd22-458a-8089-141353bce9e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:56.598285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-63398151-dd22-458a-8089-141353bce9e0 None None] MFA Rules not processed for user `a28d6bcc3a664d498d07890e2cab7f1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:56.599511 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-63398151-dd22-458a-8089-141353bce9e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:56.664651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1438/2867] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:56 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:57:56.675192 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14e4203-4c2a-4a96-9184-a0ec581c7579 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:57:56.676887 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-c14e4203-4c2a-4a96-9184-a0ec581c7579 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:56.704611 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:57:56.735995 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14e4203-4c2a-4a96-9184-a0ec581c7579 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:57:56.740676 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd58d425-de0a-4257-a3ec-81a61fd929d5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:57:56.741313 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd58d425-de0a-4257-a3ec-81a61fd929d5 service cinder] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:56.741606 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd58d425-de0a-4257-a3ec-81a61fd929d5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:56.742493 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd58d425-de0a-4257-a3ec-81a61fd929d5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:56.751320 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd58d425-de0a-4257-a3ec-81a61fd929d5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:57:56.752827 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd58d425-de0a-4257-a3ec-81a61fd929d5 service cinder] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:57:56.761599 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1430/2868] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:57:56 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:57:58.527702 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28af1f4-cf5a-43da-9b28-8b35a37cac8f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:57:58.528245 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28af1f4-cf5a-43da-9b28-8b35a37cac8f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:57:58.528371 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28af1f4-cf5a-43da-9b28-8b35a37cac8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:57:58.549588 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-c28af1f4-cf5a-43da-9b28-8b35a37cac8f None None] MFA Rules not processed for user `a28d6bcc3a664d498d07890e2cab7f1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:57:58.550900 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c28af1f4-cf5a-43da-9b28-8b35a37cac8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:57:58.558489 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1439/2869] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:57:58 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:58:06.786769 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77007a7d-039a-472d-ab5c-3873c15dd792 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:06.787336 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77007a7d-039a-472d-ab5c-3873c15dd792 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:06.787491 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77007a7d-039a-472d-ab5c-3873c15dd792 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:06.806677 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-77007a7d-039a-472d-ab5c-3873c15dd792 None None] MFA Rules not processed for user `6794c30b34984af2b3eb6c0902a6157d`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:58:06.808765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-77007a7d-039a-472d-ab5c-3873c15dd792 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:06.862711 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1431/2870] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:58:06 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:58:08.544068 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:58:12.037998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28af1f4-cf5a-43da-9b28-8b35a37cac8f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:12.039622 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c28af1f4-cf5a-43da-9b28-8b35a37cac8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:12.098693 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28af1f4-cf5a-43da-9b28-8b35a37cac8f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:12.105370 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1159a9b-3c42-4e8d-9475-63841f62b99c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:12.105985 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1159a9b-3c42-4e8d-9475-63841f62b99c service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:12.106556 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1159a9b-3c42-4e8d-9475-63841f62b99c service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:12.106834 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1159a9b-3c42-4e8d-9475-63841f62b99c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:12.108663 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b1159a9b-3c42-4e8d-9475-63841f62b99c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:12.193904 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1159a9b-3c42-4e8d-9475-63841f62b99c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:12.241873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1159a9b-3c42-4e8d-9475-63841f62b99c service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:12.241873 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1440/2871] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:58:12 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:58:13.481597 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77007a7d-039a-472d-ab5c-3873c15dd792 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:13.483070 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-77007a7d-039a-472d-ab5c-3873c15dd792 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:13.495437 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77007a7d-039a-472d-ab5c-3873c15dd792 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:13.499814 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe8e97d-d175-4e27-8896-1ae0d351bc5e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:13.500511 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe8e97d-d175-4e27-8896-1ae0d351bc5e service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:13.500795 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe8e97d-d175-4e27-8896-1ae0d351bc5e service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:13.501069 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe8e97d-d175-4e27-8896-1ae0d351bc5e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:13.502772 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-9fe8e97d-d175-4e27-8896-1ae0d351bc5e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:13.516398 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe8e97d-d175-4e27-8896-1ae0d351bc5e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:13.517846 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe8e97d-d175-4e27-8896-1ae0d351bc5e service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:13.529206 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1432/2872] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:58:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:58:17.972517 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f97dba7-f418-4d19-a633-afcc667ec919 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:17.972997 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f97dba7-f418-4d19-a633-afcc667ec919 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:17.973104 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f97dba7-f418-4d19-a633-afcc667ec919 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:17.993238 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1f97dba7-f418-4d19-a633-afcc667ec919 None None] MFA Rules not processed for user `d40810f61ead4c3da414194b325da80a`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:58:17.994419 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1f97dba7-f418-4d19-a633-afcc667ec919 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:18.049723 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1441/2873] 10.222.0.49 () {58 vars in 1009 bytes} [Fri Sep 16 03:58:17 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:58:18.231935 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe8e97d-d175-4e27-8896-1ae0d351bc5e service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:18.233979 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-9fe8e97d-d175-4e27-8896-1ae0d351bc5e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:18.302291 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe8e97d-d175-4e27-8896-1ae0d351bc5e service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:18.307400 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3235285-85cc-40ec-a519-a5b2b19e6cad service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa30da9972e140d7905c522576681429', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:18.308221 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3235285-85cc-40ec-a519-a5b2b19e6cad service swift] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:18.308612 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3235285-85cc-40ec-a519-a5b2b19e6cad service swift] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:18.308895 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3235285-85cc-40ec-a519-a5b2b19e6cad service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:18.317894 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3235285-85cc-40ec-a519-a5b2b19e6cad service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:18.319557 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3235285-85cc-40ec-a519-a5b2b19e6cad service swift] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:18.329199 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1433/2874] 10.222.0.49 () {60 vars in 1384 bytes} [Fri Sep 16 03:58:18 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 03:58:19.198363 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdb9ec1-2f98-4fc1-88df-e1ec04041d08 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:19.201931 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdb9ec1-2f98-4fc1-88df-e1ec04041d08 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:19.201931 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdb9ec1-2f98-4fc1-88df-e1ec04041d08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:19.220106 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7bdb9ec1-2f98-4fc1-88df-e1ec04041d08 None None] MFA Rules not processed for user `d40810f61ead4c3da414194b325da80a`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:58:19.221304 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7bdb9ec1-2f98-4fc1-88df-e1ec04041d08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:19.229057 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1442/2875] 10.222.0.49 () {58 vars in 967 bytes} [Fri Sep 16 03:58:19 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:58:19.242438 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3235285-85cc-40ec-a519-a5b2b19e6cad service swift] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:19.254346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3235285-85cc-40ec-a519-a5b2b19e6cad service swift] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:19.259785 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf4e5c4-ca91-4a5e-ba28-708f90aa99b2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa30da9972e140d7905c522576681429', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:19.260438 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf4e5c4-ca91-4a5e-ba28-708f90aa99b2 service swift] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:19.260716 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf4e5c4-ca91-4a5e-ba28-708f90aa99b2 service swift] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:19.260988 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf4e5c4-ca91-4a5e-ba28-708f90aa99b2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:19.269548 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf4e5c4-ca91-4a5e-ba28-708f90aa99b2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:19.271137 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf4e5c4-ca91-4a5e-ba28-708f90aa99b2 service swift] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:19.279338 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1434/2876] 10.222.0.49 () {60 vars in 1384 bytes} [Fri Sep 16 03:58:19 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 03:58:19.992373 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082a2c17-e300-41dc-948d-a6de96379d20 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:19.992663 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082a2c17-e300-41dc-948d-a6de96379d20 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:19.992942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082a2c17-e300-41dc-948d-a6de96379d20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:20.012601 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-082a2c17-e300-41dc-948d-a6de96379d20 None None] MFA Rules not processed for user `d40810f61ead4c3da414194b325da80a`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:58:20.013783 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-082a2c17-e300-41dc-948d-a6de96379d20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:20.021307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1443/2877] 10.222.0.49 () {58 vars in 1009 bytes} [Fri Sep 16 03:58:19 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:58:20.033245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf4e5c4-ca91-4a5e-ba28-708f90aa99b2 service swift] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:20.043791 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf4e5c4-ca91-4a5e-ba28-708f90aa99b2 service swift] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:20.049639 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b670bc-4430-43b4-9f46-de1284565e9b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa30da9972e140d7905c522576681429', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:20.050261 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b670bc-4430-43b4-9f46-de1284565e9b service swift] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:20.050544 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b670bc-4430-43b4-9f46-de1284565e9b service swift] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:20.050819 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b670bc-4430-43b4-9f46-de1284565e9b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:20.058964 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b670bc-4430-43b4-9f46-de1284565e9b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:20.060800 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b670bc-4430-43b4-9f46-de1284565e9b service swift] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:20.072211 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1435/2878] 10.222.0.49 () {60 vars in 1384 bytes} [Fri Sep 16 03:58:20 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 03:58:20.757613 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de71875-aed7-4ab2-a27a-6de3c5e9692c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:20.758096 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de71875-aed7-4ab2-a27a-6de3c5e9692c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:20.758194 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de71875-aed7-4ab2-a27a-6de3c5e9692c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:20.777719 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-3de71875-aed7-4ab2-a27a-6de3c5e9692c None None] MFA Rules not processed for user `d40810f61ead4c3da414194b325da80a`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:58:20.778891 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3de71875-aed7-4ab2-a27a-6de3c5e9692c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:20.786621 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1444/2879] 10.222.0.49 () {58 vars in 967 bytes} [Fri Sep 16 03:58:20 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:58:20.798658 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b670bc-4430-43b4-9f46-de1284565e9b service swift] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:20.808955 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b670bc-4430-43b4-9f46-de1284565e9b service swift] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:20.814280 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a534e48-753b-4909-8917-df52fddfec8c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa30da9972e140d7905c522576681429', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:20.814866 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a534e48-753b-4909-8917-df52fddfec8c service swift] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:20.815176 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a534e48-753b-4909-8917-df52fddfec8c service swift] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:20.815447 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a534e48-753b-4909-8917-df52fddfec8c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:20.823159 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a534e48-753b-4909-8917-df52fddfec8c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:20.824714 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a534e48-753b-4909-8917-df52fddfec8c service swift] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:20.833434 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1436/2880] 10.222.0.49 () {60 vars in 1384 bytes} [Fri Sep 16 03:58:20 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 03:58:23.556745 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de71875-aed7-4ab2-a27a-6de3c5e9692c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:23.586206 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:58:23.618313 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de71875-aed7-4ab2-a27a-6de3c5e9692c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:23.622802 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9257a206-7647-4434-ae14-24fc8653f0e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:23.623509 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9257a206-7647-4434-ae14-24fc8653f0e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:23.623797 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9257a206-7647-4434-ae14-24fc8653f0e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:23.624179 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9257a206-7647-4434-ae14-24fc8653f0e6 admin admin] PATH_INFO: `/v3/users/a28d6bcc3a664d498d07890e2cab7f1a` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:23.625538 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9257a206-7647-4434-ae14-24fc8653f0e6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a28d6bcc3a664d498d07890e2cab7f1a)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:23.628071 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9257a206-7647-4434-ae14-24fc8653f0e6 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:23.895456 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9257a206-7647-4434-ae14-24fc8653f0e6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a28d6bcc3a664d498d07890e2cab7f1a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:58:23.907903 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9257a206-7647-4434-ae14-24fc8653f0e6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a28d6bcc3a664d498d07890e2cab7f1a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:58:23.984523 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9257a206-7647-4434-ae14-24fc8653f0e6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a28d6bcc3a664d498d07890e2cab7f1a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:58:23.985139 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9257a206-7647-4434-ae14-24fc8653f0e6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a28d6bcc3a664d498d07890e2cab7f1a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:58:23.992499 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9257a206-7647-4434-ae14-24fc8653f0e6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a28d6bcc3a664d498d07890e2cab7f1a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:58:23.998323 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1445/2881] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:58:23 2022] DELETE /identity/v3/users/a28d6bcc3a664d498d07890e2cab7f1a => generated 0 bytes in 442 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:58:24.003488 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80193dba-00ab-49fb-9389-45d12601b7c2 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:24.003961 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80193dba-00ab-49fb-9389-45d12601b7c2 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:24.004124 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80193dba-00ab-49fb-9389-45d12601b7c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:24.018102 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:58:24.023561 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-80193dba-00ab-49fb-9389-45d12601b7c2 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:58:24.024735 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-80193dba-00ab-49fb-9389-45d12601b7c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:24.089141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1437/2882] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:58:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:58:24.099940 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9257a206-7647-4434-ae14-24fc8653f0e6 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:24.101272 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9257a206-7647-4434-ae14-24fc8653f0e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:24.160075 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9257a206-7647-4434-ae14-24fc8653f0e6 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:24.163502 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335411ff-94f0-48a4-bc6a-20a0c70d48b2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:24.164145 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335411ff-94f0-48a4-bc6a-20a0c70d48b2 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:24.164425 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335411ff-94f0-48a4-bc6a-20a0c70d48b2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:24.164696 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335411ff-94f0-48a4-bc6a-20a0c70d48b2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:24.175258 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335411ff-94f0-48a4-bc6a-20a0c70d48b2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:24.176806 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335411ff-94f0-48a4-bc6a-20a0c70d48b2 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:24.187589 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1446/2883] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:58:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:58:24.469443 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80193dba-00ab-49fb-9389-45d12601b7c2 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:24.470938 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-80193dba-00ab-49fb-9389-45d12601b7c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:24.483090 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80193dba-00ab-49fb-9389-45d12601b7c2 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:24.488763 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a487dc5-d626-47c9-8243-561d248e19ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:24.489410 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a487dc5-d626-47c9-8243-561d248e19ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:24.489831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a487dc5-d626-47c9-8243-561d248e19ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:24.490130 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a487dc5-d626-47c9-8243-561d248e19ac admin admin] PATH_INFO: `/v3/projects/548a2763e5c146c783f15114b4070a0d` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:24.491288 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a487dc5-d626-47c9-8243-561d248e19ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=548a2763e5c146c783f15114b4070a0d)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:24.497449 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a487dc5-d626-47c9-8243-561d248e19ac admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:24.617953 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-4a487dc5-d626-47c9-8243-561d248e19ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '548a2763e5c146c783f15114b4070a0d', 'request_id': 'req-4a487dc5-d626-47c9-8243-561d248e19ac'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:58:24.721675 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-4a487dc5-d626-47c9-8243-561d248e19ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '548a2763e5c146c783f15114b4070a0d', 'request_id': 'req-4a487dc5-d626-47c9-8243-561d248e19ac'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:58:24.728084 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-4a487dc5-d626-47c9-8243-561d248e19ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '548a2763e5c146c783f15114b4070a0d', 'request_id': 'req-4a487dc5-d626-47c9-8243-561d248e19ac'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:58:24.740124 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-4a487dc5-d626-47c9-8243-561d248e19ac admin admin] The token cache is being invalidate because project 548a2763e5c146c783f15114b4070a0d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:58:24.740418 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-4a487dc5-d626-47c9-8243-561d248e19ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:58:24.741073 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-4a487dc5-d626-47c9-8243-561d248e19ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:58:24.742569 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1438/2884] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:58:24 2022] DELETE /identity/v3/projects/548a2763e5c146c783f15114b4070a0d => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:58:24.747549 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82d7b42-cd86-49ed-9555-2edafd4c4572 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:24.748079 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82d7b42-cd86-49ed-9555-2edafd4c4572 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:24.748204 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82d7b42-cd86-49ed-9555-2edafd4c4572 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:24.769359 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e82d7b42-cd86-49ed-9555-2edafd4c4572 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:58:24.770591 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e82d7b42-cd86-49ed-9555-2edafd4c4572 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:24.815898 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:58:24.837414 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1447/2885] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:58:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:58:24.845520 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d7a9c6-54c4-40e8-9de1-f69dc8282fa5 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:24.845892 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d7a9c6-54c4-40e8-9de1-f69dc8282fa5 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:24.846182 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d7a9c6-54c4-40e8-9de1-f69dc8282fa5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:24.867483 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-28d7a9c6-54c4-40e8-9de1-f69dc8282fa5 None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:58:24.875990 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-28d7a9c6-54c4-40e8-9de1-f69dc8282fa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:24.877460 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1439/2886] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:58:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:58:24.881714 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82d7b42-cd86-49ed-9555-2edafd4c4572 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:24.891885 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82d7b42-cd86-49ed-9555-2edafd4c4572 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:24.896510 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39a874b-ff4c-4583-b65d-9d32cf97000e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:24.898962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39a874b-ff4c-4583-b65d-9d32cf97000e admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:24.898962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39a874b-ff4c-4583-b65d-9d32cf97000e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:24.898962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39a874b-ff4c-4583-b65d-9d32cf97000e admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:24.898962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39a874b-ff4c-4583-b65d-9d32cf97000e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 03:58:24.898962 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39a874b-ff4c-4583-b65d-9d32cf97000e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:24.900219 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39a874b-ff4c-4583-b65d-9d32cf97000e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:24.911516 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1448/2887] 10.222.0.49 () {58 vars in 1153 bytes} [Fri Sep 16 03:58:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 03:58:24.915716 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d7a9c6-54c4-40e8-9de1-f69dc8282fa5 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:24.924844 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d7a9c6-54c4-40e8-9de1-f69dc8282fa5 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:24.929563 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628aa125-127a-4119-821c-5fedbbcc7961 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:24.930157 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628aa125-127a-4119-821c-5fedbbcc7961 admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:24.930439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628aa125-127a-4119-821c-5fedbbcc7961 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:24.930716 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628aa125-127a-4119-821c-5fedbbcc7961 admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:24.931382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628aa125-127a-4119-821c-5fedbbcc7961 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:24.933305 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628aa125-127a-4119-821c-5fedbbcc7961 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:25.029144 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1440/2888] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:58:24 2022] POST /identity/v3/projects => generated 379 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:58:25.030257 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39a874b-ff4c-4583-b65d-9d32cf97000e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:25.040824 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39a874b-ff4c-4583-b65d-9d32cf97000e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:25.100620 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1aa198-5b58-4971-9475-564fab801b93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:25.101228 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1aa198-5b58-4971-9475-564fab801b93 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:25.101513 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1aa198-5b58-4971-9475-564fab801b93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:25.101795 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1aa198-5b58-4971-9475-564fab801b93 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:25.102426 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1aa198-5b58-4971-9475-564fab801b93 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:25.104551 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1aa198-5b58-4971-9475-564fab801b93 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:25.253739 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1449/2889] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:58:25 2022] POST /identity/v3/users => generated 290 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:58:25.257808 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628aa125-127a-4119-821c-5fedbbcc7961 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:25.269975 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628aa125-127a-4119-821c-5fedbbcc7961 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:25.275102 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bebbb1f-12b5-40c3-823c-c919f131cbbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:25.275831 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bebbb1f-12b5-40c3-823c-c919f131cbbf admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:25.276251 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bebbb1f-12b5-40c3-823c-c919f131cbbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:25.276625 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bebbb1f-12b5-40c3-823c-c919f131cbbf admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:25.277382 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bebbb1f-12b5-40c3-823c-c919f131cbbf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:25.279446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bebbb1f-12b5-40c3-823c-c919f131cbbf admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:25.291950 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1441/2890] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:58:25 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:58:25.300380 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1aa198-5b58-4971-9475-564fab801b93 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:25.312760 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1aa198-5b58-4971-9475-564fab801b93 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:25.318661 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288cb6e6-a2c9-4c88-a91d-b73dc7da9893 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:25.319654 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288cb6e6-a2c9-4c88-a91d-b73dc7da9893 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:25.320134 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288cb6e6-a2c9-4c88-a91d-b73dc7da9893 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:25.320529 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288cb6e6-a2c9-4c88-a91d-b73dc7da9893 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:25.321255 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288cb6e6-a2c9-4c88-a91d-b73dc7da9893 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:25.323957 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288cb6e6-a2c9-4c88-a91d-b73dc7da9893 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:25.336049 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1450/2891] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:58:25 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:58:25.340245 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bebbb1f-12b5-40c3-823c-c919f131cbbf admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:25.350747 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bebbb1f-12b5-40c3-823c-c919f131cbbf admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:25.355592 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a3480a-29c7-46f5-96f3-b3cdb66dc32a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:25.356256 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a3480a-29c7-46f5-96f3-b3cdb66dc32a admin admin] REQUEST_METHOD: `PUT` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:25.356537 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a3480a-29c7-46f5-96f3-b3cdb66dc32a admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:25.356816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a3480a-29c7-46f5-96f3-b3cdb66dc32a admin admin] PATH_INFO: `/v3/projects/42406ac5a18945dea55e37a91e9f5530/users/d8b2c15cd31544069f0888a04bfe0c63/roles/589e1f1333a042b7a00c53384a2f5426` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:25.372758 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a3480a-29c7-46f5-96f3-b3cdb66dc32a admin admin] RBAC: Authorizing `identity:create_grant(project_id=42406ac5a18945dea55e37a91e9f5530, user_id=d8b2c15cd31544069f0888a04bfe0c63, role_id=589e1f1333a042b7a00c53384a2f5426)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:25.376414 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a3480a-29c7-46f5-96f3-b3cdb66dc32a admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:25.530595 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1442/2892] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:58:25 2022] PUT /identity/v3/projects/42406ac5a18945dea55e37a91e9f5530/users/d8b2c15cd31544069f0888a04bfe0c63/roles/589e1f1333a042b7a00c53384a2f5426 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:58:25.535937 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad9c52a-cda2-47c1-b98e-0cf8a4ca065f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:25.536224 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad9c52a-cda2-47c1-b98e-0cf8a4ca065f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:25.540761 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad9c52a-cda2-47c1-b98e-0cf8a4ca065f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:25.556577 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-cad9c52a-cda2-47c1-b98e-0cf8a4ca065f None None] MFA Rules not processed for user `3e71cde4d415471495fcc3441fcdfc54`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:58:25.557829 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cad9c52a-cda2-47c1-b98e-0cf8a4ca065f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:25.618238 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1451/2893] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:58:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:58:25.628672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a3480a-29c7-46f5-96f3-b3cdb66dc32a admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:25.631255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-15a3480a-29c7-46f5-96f3-b3cdb66dc32a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:25.663272 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:58:25.690263 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a3480a-29c7-46f5-96f3-b3cdb66dc32a admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:25.693842 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4ff85d-5c2a-4b62-aed4-4e3b6a154d24 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:25.694442 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4ff85d-5c2a-4b62-aed4-4e3b6a154d24 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:25.694718 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4ff85d-5c2a-4b62-aed4-4e3b6a154d24 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:25.695087 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4ff85d-5c2a-4b62-aed4-4e3b6a154d24 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:25.703963 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb4ff85d-5c2a-4b62-aed4-4e3b6a154d24 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:25.705385 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb4ff85d-5c2a-4b62-aed4-4e3b6a154d24 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:25.714877 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1443/2894] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:58:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:58:31.332600 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa128fb-8b75-45ca-ac24-aa6cf2aa4846 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:31.333133 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa128fb-8b75-45ca-ac24-aa6cf2aa4846 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:31.333133 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa128fb-8b75-45ca-ac24-aa6cf2aa4846 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:31.369459 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2fa128fb-8b75-45ca-ac24-aa6cf2aa4846 None None] MFA Rules not processed for user `d8b2c15cd31544069f0888a04bfe0c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:58:31.370649 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2fa128fb-8b75-45ca-ac24-aa6cf2aa4846 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:31.418494 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:58:31.440710 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1452/2895] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:58:31 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:58:31.445302 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4ff85d-5c2a-4b62-aed4-4e3b6a154d24 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:31.455817 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4ff85d-5c2a-4b62-aed4-4e3b6a154d24 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:31.460482 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf727a4-8239-4a6e-99ff-f889c14af8ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:31.461092 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf727a4-8239-4a6e-99ff-f889c14af8ab admin admin] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:31.461371 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf727a4-8239-4a6e-99ff-f889c14af8ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:31.461646 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf727a4-8239-4a6e-99ff-f889c14af8ab admin admin] PATH_INFO: `/v3/projects` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:31.462352 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf727a4-8239-4a6e-99ff-f889c14af8ab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:31.464357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf727a4-8239-4a6e-99ff-f889c14af8ab admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:31.564293 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1444/2896] 10.222.0.49 () {60 vars in 1153 bytes} [Fri Sep 16 03:58:31 2022] POST /identity/v3/projects => generated 375 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 03:58:31.568349 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa128fb-8b75-45ca-ac24-aa6cf2aa4846 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:31.580431 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa128fb-8b75-45ca-ac24-aa6cf2aa4846 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:31.639391 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82642b22-6929-4329-9766-d90d903ab12d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:31.640866 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82642b22-6929-4329-9766-d90d903ab12d admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:31.641314 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82642b22-6929-4329-9766-d90d903ab12d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:31.641857 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82642b22-6929-4329-9766-d90d903ab12d admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:31.642638 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82642b22-6929-4329-9766-d90d903ab12d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:31.644724 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82642b22-6929-4329-9766-d90d903ab12d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:31.761857 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1453/2897] 10.222.0.49 () {60 vars in 1144 bytes} [Fri Sep 16 03:58:31 2022] POST /identity/v3/users => generated 294 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 03:58:31.766614 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf727a4-8239-4a6e-99ff-f889c14af8ab admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:31.776825 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf727a4-8239-4a6e-99ff-f889c14af8ab admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:31.781539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772ee210-9623-46c0-8118-f9175ef06755 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:31.782141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772ee210-9623-46c0-8118-f9175ef06755 admin admin] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:31.782427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772ee210-9623-46c0-8118-f9175ef06755 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:31.782704 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772ee210-9623-46c0-8118-f9175ef06755 admin admin] PATH_INFO: `/v3/roles` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:31.783383 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772ee210-9623-46c0-8118-f9175ef06755 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:31.784977 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772ee210-9623-46c0-8118-f9175ef06755 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:31.795461 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1445/2898] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:58:31 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 03:58:31.800756 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82642b22-6929-4329-9766-d90d903ab12d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:31.810993 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82642b22-6929-4329-9766-d90d903ab12d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:31.816361 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063f5cb6-d205-43aa-af2f-c07a5e768d57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:31.824239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063f5cb6-d205-43aa-af2f-c07a5e768d57 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:31.824239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063f5cb6-d205-43aa-af2f-c07a5e768d57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:31.824239 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063f5cb6-d205-43aa-af2f-c07a5e768d57 admin admin] PATH_INFO: `/v3/projects/a1dedaa8d09543c1a896888969b33f28/users/1ddde60407934d77974472db40bf70cf/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:31.831281 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05364d3-8663-4e1a-a779-55d7f2eb88b8 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:31.831662 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05364d3-8663-4e1a-a779-55d7f2eb88b8 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:31.831841 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05364d3-8663-4e1a-a779-55d7f2eb88b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:31.834995 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063f5cb6-d205-43aa-af2f-c07a5e768d57 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a1dedaa8d09543c1a896888969b33f28, user_id=1ddde60407934d77974472db40bf70cf, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:31.837664 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063f5cb6-d205-43aa-af2f-c07a5e768d57 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:31.850493 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-a05364d3-8663-4e1a-a779-55d7f2eb88b8 None None] MFA Rules not processed for user `d40810f61ead4c3da414194b325da80a`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:58:31.851706 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a05364d3-8663-4e1a-a779-55d7f2eb88b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:31.907371 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1446/2899] 10.222.0.49 () {58 vars in 1009 bytes} [Fri Sep 16 03:58:31 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 03:58:31.919818 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05364d3-8663-4e1a-a779-55d7f2eb88b8 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:31.921731 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-a05364d3-8663-4e1a-a779-55d7f2eb88b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:31.933164 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1454/2900] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:58:31 2022] PUT /identity/v3/projects/a1dedaa8d09543c1a896888969b33f28/users/1ddde60407934d77974472db40bf70cf/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:58:31.937925 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063f5cb6-d205-43aa-af2f-c07a5e768d57 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:31.996052 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05364d3-8663-4e1a-a779-55d7f2eb88b8 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:32.004883 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063f5cb6-d205-43aa-af2f-c07a5e768d57 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:32.009916 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3172d52-817b-4975-8de5-a7a75be2a8f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:32.010747 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3172d52-817b-4975-8de5-a7a75be2a8f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:32.011557 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3172d52-817b-4975-8de5-a7a75be2a8f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:32.011897 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3172d52-817b-4975-8de5-a7a75be2a8f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:32.012608 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3172d52-817b-4975-8de5-a7a75be2a8f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:32.014364 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3172d52-817b-4975-8de5-a7a75be2a8f4 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:32.026936 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1455/2901] 10.222.0.49 () {58 vars in 1122 bytes} [Fri Sep 16 03:58:31 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 03:58:32.030928 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3172d52-817b-4975-8de5-a7a75be2a8f4 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:32.040651 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3172d52-817b-4975-8de5-a7a75be2a8f4 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:32.044385 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:58:32.046058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2a12ec-dc68-4ee6-ae12-952d1edb9263 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:32.046794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2a12ec-dc68-4ee6-ae12-952d1edb9263 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:32.047110 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2a12ec-dc68-4ee6-ae12-952d1edb9263 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:32.047391 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2a12ec-dc68-4ee6-ae12-952d1edb9263 admin admin] PATH_INFO: `/v3/projects/a1dedaa8d09543c1a896888969b33f28/users/1ddde60407934d77974472db40bf70cf/roles/89c4b3a971224d468d33d2dc411230a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:32.049339 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2a12ec-dc68-4ee6-ae12-952d1edb9263 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a1dedaa8d09543c1a896888969b33f28, user_id=1ddde60407934d77974472db40bf70cf, role_id=89c4b3a971224d468d33d2dc411230a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:32.052034 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2a12ec-dc68-4ee6-ae12-952d1edb9263 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:32.063833 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1456/2902] 10.222.0.49 () {60 vars in 1483 bytes} [Fri Sep 16 03:58:32 2022] PUT /identity/v3/projects/a1dedaa8d09543c1a896888969b33f28/users/1ddde60407934d77974472db40bf70cf/roles/89c4b3a971224d468d33d2dc411230a2 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 03:58:32.070220 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3877a440-a302-47c4-800e-c5ceeabba443 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa30da9972e140d7905c522576681429', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:32.070943 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3877a440-a302-47c4-800e-c5ceeabba443 service swift] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:32.071314 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3877a440-a302-47c4-800e-c5ceeabba443 service swift] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:32.071666 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3877a440-a302-47c4-800e-c5ceeabba443 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:32.133760 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3877a440-a302-47c4-800e-c5ceeabba443 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:32.135395 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3877a440-a302-47c4-800e-c5ceeabba443 service swift] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:32.149359 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1447/2903] 10.222.0.49 () {60 vars in 1384 bytes} [Fri Sep 16 03:58:31 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 16 03:58:39.580777 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dffe53-e0cd-4d09-a2bf-75a5ea17b71f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:39.581285 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dffe53-e0cd-4d09-a2bf-75a5ea17b71f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:39.581412 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dffe53-e0cd-4d09-a2bf-75a5ea17b71f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:39.626018 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a4dffe53-e0cd-4d09-a2bf-75a5ea17b71f None None] MFA Rules not processed for user `1ddde60407934d77974472db40bf70cf`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:58:39.627207 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a4dffe53-e0cd-4d09-a2bf-75a5ea17b71f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:39.652782 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:58:39.721111 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1457/2904] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:58:39 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:58:39.739409 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3877a440-a302-47c4-800e-c5ceeabba443 service swift] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:39.742728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3877a440-a302-47c4-800e-c5ceeabba443 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:39.852773 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3877a440-a302-47c4-800e-c5ceeabba443 service swift] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:39.956672 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3da985-77a1-47c8-97ff-6645f8e82333 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:39.957605 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3da985-77a1-47c8-97ff-6645f8e82333 service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:39.958067 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3da985-77a1-47c8-97ff-6645f8e82333 service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:39.958526 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3da985-77a1-47c8-97ff-6645f8e82333 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:39.992713 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=100 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:58:40.035834 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3da985-77a1-47c8-97ff-6645f8e82333 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:40.037499 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3da985-77a1-47c8-97ff-6645f8e82333 service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:40.047930 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1448/2905] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:58:39 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:58:49.737182 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=19 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:58:50.058065 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=18 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:58:55.831357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6719453e-0ca7-455e-89ed-fbd24c46b6bf None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:55.831826 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6719453e-0ca7-455e-89ed-fbd24c46b6bf None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:55.831927 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6719453e-0ca7-455e-89ed-fbd24c46b6bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:55.881705 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-6719453e-0ca7-455e-89ed-fbd24c46b6bf None None] MFA Rules not processed for user `d40810f61ead4c3da414194b325da80a`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:58:55.882942 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6719453e-0ca7-455e-89ed-fbd24c46b6bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:55.946863 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1458/2906] 10.222.0.49 () {58 vars in 967 bytes} [Fri Sep 16 03:58:55 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:58:55.959028 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3da985-77a1-47c8-97ff-6645f8e82333 service nova] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:56.036549 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3da985-77a1-47c8-97ff-6645f8e82333 service nova] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:56.041617 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f21a4c-c981-4ddc-9c3d-702d896adfe2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'aa30da9972e140d7905c522576681429', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:56.042212 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f21a4c-c981-4ddc-9c3d-702d896adfe2 service swift] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:56.042492 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f21a4c-c981-4ddc-9c3d-702d896adfe2 service swift] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:56.042765 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f21a4c-c981-4ddc-9c3d-702d896adfe2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:56.051236 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f21a4c-c981-4ddc-9c3d-702d896adfe2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:56.052838 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f21a4c-c981-4ddc-9c3d-702d896adfe2 service swift] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:56.060718 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1449/2907] 10.222.0.49 () {60 vars in 1384 bytes} [Fri Sep 16 03:58:55 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 03:58:59.228438 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6719453e-0ca7-455e-89ed-fbd24c46b6bf None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:59.230570 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6719453e-0ca7-455e-89ed-fbd24c46b6bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:58:59.307521 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6719453e-0ca7-455e-89ed-fbd24c46b6bf None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:59.312348 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb5f1ec-dc3d-45d7-b8e4-a785129cb8d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:59.313473 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb5f1ec-dc3d-45d7-b8e4-a785129cb8d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:59.313750 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb5f1ec-dc3d-45d7-b8e4-a785129cb8d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:59.314022 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb5f1ec-dc3d-45d7-b8e4-a785129cb8d9 admin admin] PATH_INFO: `/v3/users/6794c30b34984af2b3eb6c0902a6157d` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:59.315242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb5f1ec-dc3d-45d7-b8e4-a785129cb8d9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6794c30b34984af2b3eb6c0902a6157d)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:59.317137 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb5f1ec-dc3d-45d7-b8e4-a785129cb8d9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:58:59.506669 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fbb5f1ec-dc3d-45d7-b8e4-a785129cb8d9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6794c30b34984af2b3eb6c0902a6157d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:58:59.518949 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fbb5f1ec-dc3d-45d7-b8e4-a785129cb8d9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6794c30b34984af2b3eb6c0902a6157d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:58:59.602332 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fbb5f1ec-dc3d-45d7-b8e4-a785129cb8d9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6794c30b34984af2b3eb6c0902a6157d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:58:59.603097 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fbb5f1ec-dc3d-45d7-b8e4-a785129cb8d9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6794c30b34984af2b3eb6c0902a6157d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:58:59.605579 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:58:59.612007 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fbb5f1ec-dc3d-45d7-b8e4-a785129cb8d9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6794c30b34984af2b3eb6c0902a6157d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:58:59.617907 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1459/2908] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 03:58:59 2022] DELETE /identity/v3/users/6794c30b34984af2b3eb6c0902a6157d => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:58:59.856251 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f21a4c-c981-4ddc-9c3d-702d896adfe2 service swift] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:58:59.878935 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f21a4c-c981-4ddc-9c3d-702d896adfe2 service swift] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:58:59.945508 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf06530-bcae-4f63-93f1-9310bd2bfe80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:58:59.948787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf06530-bcae-4f63-93f1-9310bd2bfe80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:58:59.948787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf06530-bcae-4f63-93f1-9310bd2bfe80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:58:59.948787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf06530-bcae-4f63-93f1-9310bd2bfe80 admin admin] PATH_INFO: `/v3/projects/ada320ab9e1644f29fd613c173ffe0a8` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:58:59.948787 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf06530-bcae-4f63-93f1-9310bd2bfe80 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ada320ab9e1644f29fd613c173ffe0a8)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:58:59.949717 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf06530-bcae-4f63-93f1-9310bd2bfe80 admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:59:00.081185 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-bcf06530-bcae-4f63-93f1-9310bd2bfe80 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ada320ab9e1644f29fd613c173ffe0a8', 'request_id': 'req-bcf06530-bcae-4f63-93f1-9310bd2bfe80'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:59:00.175255 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-bcf06530-bcae-4f63-93f1-9310bd2bfe80 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ada320ab9e1644f29fd613c173ffe0a8', 'request_id': 'req-bcf06530-bcae-4f63-93f1-9310bd2bfe80'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:59:00.180980 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-bcf06530-bcae-4f63-93f1-9310bd2bfe80 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ada320ab9e1644f29fd613c173ffe0a8', 'request_id': 'req-bcf06530-bcae-4f63-93f1-9310bd2bfe80'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:59:00.192485 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-bcf06530-bcae-4f63-93f1-9310bd2bfe80 admin admin] The token cache is being invalidate because project ada320ab9e1644f29fd613c173ffe0a8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 03:59:00.192773 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-bcf06530-bcae-4f63-93f1-9310bd2bfe80 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:59:00.193427 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-bcf06530-bcae-4f63-93f1-9310bd2bfe80 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 03:59:00.194966 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1450/2909] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 03:58:59 2022] DELETE /identity/v3/projects/ada320ab9e1644f29fd613c173ffe0a8 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 03:59:05.308195 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecce4f7-0aa9-4a93-ae75-16e17d99acb2 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:59:05.308761 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecce4f7-0aa9-4a93-ae75-16e17d99acb2 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:59:05.308761 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecce4f7-0aa9-4a93-ae75-16e17d99acb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:59:05.329746 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8ecce4f7-0aa9-4a93-ae75-16e17d99acb2 None None] MFA Rules not processed for user `d8b2c15cd31544069f0888a04bfe0c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 03:59:05.331037 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8ecce4f7-0aa9-4a93-ae75-16e17d99acb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:59:05.384858 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1460/2910] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 03:59:05 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 03:59:05.702559 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf06530-bcae-4f63-93f1-9310bd2bfe80 admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:59:05.703928 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-bcf06530-bcae-4f63-93f1-9310bd2bfe80 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:59:05.717699 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:59:05.760021 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf06530-bcae-4f63-93f1-9310bd2bfe80 admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:59:05.763665 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8d3678-fbb3-4244-9797-0907ce80cac0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:59:05.764259 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8d3678-fbb3-4244-9797-0907ce80cac0 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:59:05.764535 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8d3678-fbb3-4244-9797-0907ce80cac0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:59:05.764806 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8d3678-fbb3-4244-9797-0907ce80cac0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:59:05.766497 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-2c8d3678-fbb3-4244-9797-0907ce80cac0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:59:05.776486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8d3678-fbb3-4244-9797-0907ce80cac0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:59:05.777897 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8d3678-fbb3-4244-9797-0907ce80cac0 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:59:05.786307 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1451/2911] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:59:05 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:59:06.478609 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecce4f7-0aa9-4a93-ae75-16e17d99acb2 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:59:06.480319 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8ecce4f7-0aa9-4a93-ae75-16e17d99acb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:59:06.565975 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecce4f7-0aa9-4a93-ae75-16e17d99acb2 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:59:06.571358 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb84a326-6a90-487a-8703-836a43f0333a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '176da4679fcb46148da9289fccdad5c2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:59:06.571964 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb84a326-6a90-487a-8703-836a43f0333a service placement] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:59:06.572272 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb84a326-6a90-487a-8703-836a43f0333a service placement] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:59:06.572541 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb84a326-6a90-487a-8703-836a43f0333a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:59:06.574243 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fb84a326-6a90-487a-8703-836a43f0333a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:59:06.585912 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb84a326-6a90-487a-8703-836a43f0333a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:59:06.587520 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb84a326-6a90-487a-8703-836a43f0333a service placement] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:59:06.597848 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1461/2912] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:59:06 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:59:10.360910 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8d3678-fbb3-4244-9797-0907ce80cac0 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:59:10.373231 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8d3678-fbb3-4244-9797-0907ce80cac0 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:59:10.379228 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0409035a-8e0b-4a7d-95e2-25f7bdac0eb8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:59:10.380482 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0409035a-8e0b-4a7d-95e2-25f7bdac0eb8 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:59:10.380916 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0409035a-8e0b-4a7d-95e2-25f7bdac0eb8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:59:10.381357 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0409035a-8e0b-4a7d-95e2-25f7bdac0eb8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:59:10.383385 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-0409035a-8e0b-4a7d-95e2-25f7bdac0eb8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:59:10.394576 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0409035a-8e0b-4a7d-95e2-25f7bdac0eb8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:59:10.399388 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0409035a-8e0b-4a7d-95e2-25f7bdac0eb8 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:59:10.404459 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1452/2913] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:59:10 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:59:11.255552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb84a326-6a90-487a-8703-836a43f0333a service placement] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:59:11.260662 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fb84a326-6a90-487a-8703-836a43f0333a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:59:11.334123 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb84a326-6a90-487a-8703-836a43f0333a service placement] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:59:11.339877 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3b9301-05a1-42d0-8984-b2f892e12edd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:59:11.340484 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3b9301-05a1-42d0-8984-b2f892e12edd service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:59:11.340770 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3b9301-05a1-42d0-8984-b2f892e12edd service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:59:11.341045 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3b9301-05a1-42d0-8984-b2f892e12edd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:59:11.342767 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cf3b9301-05a1-42d0-8984-b2f892e12edd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:59:11.355353 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3b9301-05a1-42d0-8984-b2f892e12edd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:59:11.356937 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3b9301-05a1-42d0-8984-b2f892e12edd service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:59:11.368881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1462/2914] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:59:11 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 03:59:20.400856 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=27 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:59:21.364678 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 03:59:25.322248 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0409035a-8e0b-4a7d-95e2-25f7bdac0eb8 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 03:59:25.331346 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0409035a-8e0b-4a7d-95e2-25f7bdac0eb8 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 03:59:25.335095 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa61278-b3d2-4cb6-a2f6-6bd8777aa402 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 03:59:25.335694 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa61278-b3d2-4cb6-a2f6-6bd8777aa402 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 03:59:25.335975 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa61278-b3d2-4cb6-a2f6-6bd8777aa402 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 03:59:25.336249 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa61278-b3d2-4cb6-a2f6-6bd8777aa402 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 03:59:25.338050 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-daa61278-b3d2-4cb6-a2f6-6bd8777aa402 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 03:59:25.401749 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa61278-b3d2-4cb6-a2f6-6bd8777aa402 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 03:59:25.411236 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa61278-b3d2-4cb6-a2f6-6bd8777aa402 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 03:59:25.413409 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1453/2915] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 03:59:25 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 03:59:35.408448 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=30 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 04:01:27.134267 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3b9301-05a1-42d0-8984-b2f892e12edd service nova] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 04:01:27.147815 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3b9301-05a1-42d0-8984-b2f892e12edd service nova] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 04:01:27.153566 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7badb00-165a-4b12-babe-af91237db33f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '176da4679fcb46148da9289fccdad5c2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 04:01:27.154592 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7badb00-165a-4b12-babe-af91237db33f service placement] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 04:01:27.155242 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7badb00-165a-4b12-babe-af91237db33f service placement] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 04:01:27.155540 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7badb00-165a-4b12-babe-af91237db33f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 04:01:27.160052 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e7badb00-165a-4b12-babe-af91237db33f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 04:01:27.173038 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7badb00-165a-4b12-babe-af91237db33f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 04:01:27.174656 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7badb00-165a-4b12-babe-af91237db33f service placement] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 04:01:27.189416 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1463/2916] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 04:01:27 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 04:01:29.380880 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bce4338-d7b3-40ee-b655-e3c5da515482 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 04:01:29.381376 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bce4338-d7b3-40ee-b655-e3c5da515482 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 04:01:29.381497 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bce4338-d7b3-40ee-b655-e3c5da515482 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 04:01:29.400872 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-3bce4338-d7b3-40ee-b655-e3c5da515482 None None] MFA Rules not processed for user `d8b2c15cd31544069f0888a04bfe0c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 04:01:29.402651 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-3bce4338-d7b3-40ee-b655-e3c5da515482 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 04:01:29.412860 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1454/2917] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 04:01:29 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 04:01:37.182443 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=6 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 04:01:39.394810 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=3 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 04:01:44.406616 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7badb00-165a-4b12-babe-af91237db33f service placement] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 04:01:44.408454 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e7badb00-165a-4b12-babe-af91237db33f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 04:01:44.426606 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7badb00-165a-4b12-babe-af91237db33f service placement] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 04:01:44.432307 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3343f2a8-a36f-4488-89f7-27650ce7a00c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 04:01:44.432916 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3343f2a8-a36f-4488-89f7-27650ce7a00c service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 04:01:44.433196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3343f2a8-a36f-4488-89f7-27650ce7a00c service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 04:01:44.433469 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3343f2a8-a36f-4488-89f7-27650ce7a00c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 04:01:44.435290 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3343f2a8-a36f-4488-89f7-27650ce7a00c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 04:01:44.449333 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3343f2a8-a36f-4488-89f7-27650ce7a00c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 04:01:44.450881 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3343f2a8-a36f-4488-89f7-27650ce7a00c service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 04:01:44.462627 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1464/2918] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 04:01:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 04:01:54.457558 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=6 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 04:03:35.816342 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c5f8f5-7560-40aa-8d7b-8bdeb6b6c6a1 None None] REQUEST_METHOD: `POST` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 04:03:35.816874 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c5f8f5-7560-40aa-8d7b-8bdeb6b6c6a1 None None] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 04:03:35.816874 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c5f8f5-7560-40aa-8d7b-8bdeb6b6c6a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 04:03:35.835816 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.auth.core [None req-f5c5f8f5-7560-40aa-8d7b-8bdeb6b6c6a1 None None] MFA Rules not processed for user `d8b2c15cd31544069f0888a04bfe0c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81761) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 04:03:35.836999 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-f5c5f8f5-7560-40aa-8d7b-8bdeb6b6c6a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 04:03:35.844634 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1455/2919] 10.222.0.49 () {58 vars in 958 bytes} [Fri Sep 16 04:03:35 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 04:03:35.887966 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3343f2a8-a36f-4488-89f7-27650ce7a00c service nova] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 04:03:35.889411 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3343f2a8-a36f-4488-89f7-27650ce7a00c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 04:03:35.945449 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3343f2a8-a36f-4488-89f7-27650ce7a00c service nova] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 04:03:35.950012 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de65f432-b2ce-4b48-8ec0-0c197173112a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 04:03:35.950613 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de65f432-b2ce-4b48-8ec0-0c197173112a service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 04:03:35.950891 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de65f432-b2ce-4b48-8ec0-0c197173112a service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 04:03:35.951193 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de65f432-b2ce-4b48-8ec0-0c197173112a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 04:03:35.953009 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-de65f432-b2ce-4b48-8ec0-0c197173112a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 04:03:35.966819 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de65f432-b2ce-4b48-8ec0-0c197173112a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 04:03:35.968270 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de65f432-b2ce-4b48-8ec0-0c197173112a service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 04:03:35.979117 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1465/2920] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 04:03:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 04:03:37.023353 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c5f8f5-7560-40aa-8d7b-8bdeb6b6c6a1 None None] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 04:03:37.024904 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-f5c5f8f5-7560-40aa-8d7b-8bdeb6b6c6a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 04:03:37.038009 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c5f8f5-7560-40aa-8d7b-8bdeb6b6c6a1 None None] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 04:03:37.041791 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8eea49a-a893-45ad-962e-fce1fcfbb146 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 04:03:37.044240 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eea49a-a893-45ad-962e-fce1fcfbb146 service neutron] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 04:03:37.044240 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eea49a-a893-45ad-962e-fce1fcfbb146 service neutron] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 04:03:37.044240 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eea49a-a893-45ad-962e-fce1fcfbb146 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 04:03:37.045531 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-b8eea49a-a893-45ad-962e-fce1fcfbb146 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 04:03:37.060437 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8eea49a-a893-45ad-962e-fce1fcfbb146 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 04:03:37.062054 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8eea49a-a893-45ad-962e-fce1fcfbb146 service neutron] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 04:03:37.079071 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1456/2921] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 04:03:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 04:03:39.976629 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de65f432-b2ce-4b48-8ec0-0c197173112a service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 04:03:39.980077 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-de65f432-b2ce-4b48-8ec0-0c197173112a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 04:03:39.989663 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de65f432-b2ce-4b48-8ec0-0c197173112a service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 04:03:39.993404 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edc6ccb-7941-443a-82dd-5855441a4a47 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e6aeffd924734771bc5bf0d282c9052d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 04:03:39.994007 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edc6ccb-7941-443a-82dd-5855441a4a47 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 04:03:39.994282 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edc6ccb-7941-443a-82dd-5855441a4a47 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 04:03:39.994552 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edc6ccb-7941-443a-82dd-5855441a4a47 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 04:03:39.997259 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3edc6ccb-7941-443a-82dd-5855441a4a47 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 04:03:40.008284 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3edc6ccb-7941-443a-82dd-5855441a4a47 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 04:03:40.009635 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3edc6ccb-7941-443a-82dd-5855441a4a47 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 04:03:40.025317 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1466/2922] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 04:03:39 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 04:03:47.070422 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=9 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 04:03:50.015689 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=32 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 04:03:56.899439 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8eea49a-a893-45ad-962e-fce1fcfbb146 service neutron] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 04:03:56.911241 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8eea49a-a893-45ad-962e-fce1fcfbb146 service neutron] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 04:03:56.916985 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e162dd29-22c3-4100-bd60-c097bff9270d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 04:03:56.917581 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e162dd29-22c3-4100-bd60-c097bff9270d service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 04:03:56.917859 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e162dd29-22c3-4100-bd60-c097bff9270d service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 04:03:56.918131 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e162dd29-22c3-4100-bd60-c097bff9270d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 04:03:56.920066 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-e162dd29-22c3-4100-bd60-c097bff9270d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 04:03:56.932088 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e162dd29-22c3-4100-bd60-c097bff9270d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 04:03:56.933658 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e162dd29-22c3-4100-bd60-c097bff9270d service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 04:03:56.945051 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1457/2923] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 04:03:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 04:04:06.940162 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=6 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 04:04:33.175996 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edc6ccb-7941-443a-82dd-5855441a4a47 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 04:04:33.185192 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edc6ccb-7941-443a-82dd-5855441a4a47 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 04:04:33.189409 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d6f024-be1b-4455-bef2-3c0652a2de3a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 04:04:33.190012 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d6f024-be1b-4455-bef2-3c0652a2de3a service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 04:04:33.190293 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d6f024-be1b-4455-bef2-3c0652a2de3a service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 04:04:33.190569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d6f024-be1b-4455-bef2-3c0652a2de3a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 04:04:33.198210 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d6f024-be1b-4455-bef2-3c0652a2de3a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 04:04:33.199661 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d6f024-be1b-4455-bef2-3c0652a2de3a service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 04:04:33.208357 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1467/2924] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 04:04:33 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 33 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 04:04:43.204650 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=6 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 04:05:27.995836 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e162dd29-22c3-4100-bd60-c097bff9270d service nova] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 04:05:28.037355 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e162dd29-22c3-4100-bd60-c097bff9270d service nova] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 04:05:28.043664 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27b0694-761a-488f-956e-7aa2db8887db service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded1bcd10e62499a9aaa0ece5aa55c80', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 04:05:28.044465 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27b0694-761a-488f-956e-7aa2db8887db service nova] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 04:05:28.044745 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27b0694-761a-488f-956e-7aa2db8887db service nova] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 04:05:28.045017 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27b0694-761a-488f-956e-7aa2db8887db service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 04:05:28.054753 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27b0694-761a-488f-956e-7aa2db8887db service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 04:05:28.056347 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27b0694-761a-488f-956e-7aa2db8887db service nova] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 04:05:28.067115 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1458/2925] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 04:05:27 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 04:05:38.062354 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=20 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 04:05:41.020321 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d6f024-be1b-4455-bef2-3c0652a2de3a service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 04:05:41.058359 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d6f024-be1b-4455-bef2-3c0652a2de3a service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 04:05:41.062007 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d683fd99-6fb7-43dc-b9f7-b2cbe65deffc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f9275eeba471423e9fa2a31a6e08676d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 04:05:41.062602 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d683fd99-6fb7-43dc-b9f7-b2cbe65deffc service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 04:05:41.062884 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d683fd99-6fb7-43dc-b9f7-b2cbe65deffc service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 04:05:41.063196 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d683fd99-6fb7-43dc-b9f7-b2cbe65deffc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 04:05:41.064904 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d683fd99-6fb7-43dc-b9f7-b2cbe65deffc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 04:05:41.158506 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d683fd99-6fb7-43dc-b9f7-b2cbe65deffc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 04:05:41.159963 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d683fd99-6fb7-43dc-b9f7-b2cbe65deffc service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 04:05:41.179538 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1468/2926] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 04:05:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 04:05:44.662641 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27b0694-761a-488f-956e-7aa2db8887db service nova] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 04:05:44.664161 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-d27b0694-761a-488f-956e-7aa2db8887db service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 04:05:44.679345 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27b0694-761a-488f-956e-7aa2db8887db service nova] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 04:05:44.685490 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b823c89-4c08-4b39-be9a-9a8724f35c11 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '176da4679fcb46148da9289fccdad5c2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 04:05:44.686136 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b823c89-4c08-4b39-be9a-9a8724f35c11 service placement] REQUEST_METHOD: `GET` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 04:05:44.686438 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b823c89-4c08-4b39-be9a-9a8724f35c11 service placement] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 04:05:44.686717 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b823c89-4c08-4b39-be9a-9a8724f35c11 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 04:05:44.688486 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-8b823c89-4c08-4b39-be9a-9a8724f35c11 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 04:05:44.702141 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b823c89-4c08-4b39-be9a-9a8724f35c11 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 04:05:44.703790 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b823c89-4c08-4b39-be9a-9a8724f35c11 service placement] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 04:05:44.714428 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1459/2927] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 04:05:44 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 04:05:48.672504 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d683fd99-6fb7-43dc-b9f7-b2cbe65deffc service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 04:05:48.675163 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d683fd99-6fb7-43dc-b9f7-b2cbe65deffc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 04:05:48.688260 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d683fd99-6fb7-43dc-b9f7-b2cbe65deffc service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 04:05:48.693344 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33926709-eaa5-4239-a725-bfd8d3e65433 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 04:05:48.693970 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33926709-eaa5-4239-a725-bfd8d3e65433 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 04:05:48.694251 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33926709-eaa5-4239-a725-bfd8d3e65433 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 04:05:48.694525 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33926709-eaa5-4239-a725-bfd8d3e65433 admin admin] PATH_INFO: `/v3/users/d8b2c15cd31544069f0888a04bfe0c63` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 04:05:48.696372 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33926709-eaa5-4239-a725-bfd8d3e65433 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d8b2c15cd31544069f0888a04bfe0c63)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 04:05:48.698289 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33926709-eaa5-4239-a725-bfd8d3e65433 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 04:05:48.876851 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-33926709-eaa5-4239-a725-bfd8d3e65433 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd8b2c15cd31544069f0888a04bfe0c63'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 04:05:48.898198 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-33926709-eaa5-4239-a725-bfd8d3e65433 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd8b2c15cd31544069f0888a04bfe0c63'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 04:05:48.969033 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-33926709-eaa5-4239-a725-bfd8d3e65433 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd8b2c15cd31544069f0888a04bfe0c63'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 04:05:48.969774 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-33926709-eaa5-4239-a725-bfd8d3e65433 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd8b2c15cd31544069f0888a04bfe0c63'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 04:05:48.983644 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-33926709-eaa5-4239-a725-bfd8d3e65433 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd8b2c15cd31544069f0888a04bfe0c63'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 04:05:48.990189 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1469/2928] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 04:05:48 2022] DELETE /identity/v3/users/d8b2c15cd31544069f0888a04bfe0c63 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 04:05:49.228319 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b823c89-4c08-4b39-be9a-9a8724f35c11 service placement] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 04:05:49.239728 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b823c89-4c08-4b39-be9a-9a8724f35c11 service placement] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 04:05:49.298974 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b530d47-ab74-4aac-b7cc-383a2a384faa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 04:05:49.299600 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b530d47-ab74-4aac-b7cc-383a2a384faa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 04:05:49.299881 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b530d47-ab74-4aac-b7cc-383a2a384faa admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 04:05:49.300149 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b530d47-ab74-4aac-b7cc-383a2a384faa admin admin] PATH_INFO: `/v3/projects/42406ac5a18945dea55e37a91e9f5530` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 04:05:49.301246 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b530d47-ab74-4aac-b7cc-383a2a384faa admin admin] RBAC: Authorizing `identity:delete_project(project_id=42406ac5a18945dea55e37a91e9f5530)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 04:05:49.303151 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b530d47-ab74-4aac-b7cc-383a2a384faa admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 04:05:49.388060 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-7b530d47-ab74-4aac-b7cc-383a2a384faa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '42406ac5a18945dea55e37a91e9f5530', 'request_id': 'req-7b530d47-ab74-4aac-b7cc-383a2a384faa'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 04:05:49.448153 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-7b530d47-ab74-4aac-b7cc-383a2a384faa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '42406ac5a18945dea55e37a91e9f5530', 'request_id': 'req-7b530d47-ab74-4aac-b7cc-383a2a384faa'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 04:05:49.453840 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-7b530d47-ab74-4aac-b7cc-383a2a384faa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '42406ac5a18945dea55e37a91e9f5530', 'request_id': 'req-7b530d47-ab74-4aac-b7cc-383a2a384faa'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 04:05:49.467778 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-7b530d47-ab74-4aac-b7cc-383a2a384faa admin admin] The token cache is being invalidate because project 42406ac5a18945dea55e37a91e9f5530 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 04:05:49.468062 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-7b530d47-ab74-4aac-b7cc-383a2a384faa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 04:05:49.468702 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-7b530d47-ab74-4aac-b7cc-383a2a384faa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 04:05:49.470203 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1460/2929] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 04:05:49 2022] DELETE /identity/v3/projects/42406ac5a18945dea55e37a91e9f5530 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 04:05:49.474210 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33926709-eaa5-4239-a725-bfd8d3e65433 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 04:05:49.475697 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-33926709-eaa5-4239-a725-bfd8d3e65433 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 04:05:49.484669 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 04:05:49.490204 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33926709-eaa5-4239-a725-bfd8d3e65433 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 04:05:49.547697 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e1374d-ed87-47d9-8138-818465b03c34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 04:05:49.548325 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e1374d-ed87-47d9-8138-818465b03c34 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 04:05:49.548600 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e1374d-ed87-47d9-8138-818465b03c34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 04:05:49.548868 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e1374d-ed87-47d9-8138-818465b03c34 admin admin] PATH_INFO: `/v3/users/1ddde60407934d77974472db40bf70cf` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 04:05:49.550058 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e1374d-ed87-47d9-8138-818465b03c34 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1ddde60407934d77974472db40bf70cf)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 04:05:49.551994 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e1374d-ed87-47d9-8138-818465b03c34 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 04:05:49.767423 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d3e1374d-ed87-47d9-8138-818465b03c34 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1ddde60407934d77974472db40bf70cf'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 04:05:49.780443 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d3e1374d-ed87-47d9-8138-818465b03c34 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1ddde60407934d77974472db40bf70cf'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 04:05:49.848569 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d3e1374d-ed87-47d9-8138-818465b03c34 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1ddde60407934d77974472db40bf70cf'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 04:05:49.849181 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d3e1374d-ed87-47d9-8138-818465b03c34 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1ddde60407934d77974472db40bf70cf'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 04:05:49.856534 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d3e1374d-ed87-47d9-8138-818465b03c34 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1ddde60407934d77974472db40bf70cf'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 04:05:49.862431 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1470/2930] 10.222.0.49 () {58 vars in 1224 bytes} [Fri Sep 16 04:05:49 2022] DELETE /identity/v3/users/1ddde60407934d77974472db40bf70cf => generated 0 bytes in 389 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 04:05:50.097905 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b530d47-ab74-4aac-b7cc-383a2a384faa admin admin] Authenticating user token {{(pid=81761) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 04:05:50.099374 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.fernet_utils [None req-7b530d47-ab74-4aac-b7cc-383a2a384faa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81761) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 04:05:50.118693 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b530d47-ab74-4aac-b7cc-383a2a384faa admin admin] Validating token access rules against request {{(pid=81761) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 04:05:50.176351 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf83d7a-821c-439f-b23e-3be07c916d3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e71cde4d415471495fcc3441fcdfc54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ccdd7a44ba5049fb939dad44d5bd406e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81761) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 04:05:50.176898 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf83d7a-821c-439f-b23e-3be07c916d3c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 04:05:50.177175 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf83d7a-821c-439f-b23e-3be07c916d3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 04:05:50.177446 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf83d7a-821c-439f-b23e-3be07c916d3c admin admin] PATH_INFO: `/v3/projects/a1dedaa8d09543c1a896888969b33f28` {{(pid=81761) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 04:05:50.178539 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf83d7a-821c-439f-b23e-3be07c916d3c admin admin] RBAC: Authorizing `identity:delete_project(project_id=a1dedaa8d09543c1a896888969b33f28)` {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 04:05:50.180578 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf83d7a-821c-439f-b23e-3be07c916d3c admin admin] RBAC: Authorization granted {{(pid=81761) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 04:05:50.202536 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 04:05:50.274277 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-dcf83d7a-821c-439f-b23e-3be07c916d3c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a1dedaa8d09543c1a896888969b33f28', 'request_id': 'req-dcf83d7a-821c-439f-b23e-3be07c916d3c'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 04:05:50.364306 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-dcf83d7a-821c-439f-b23e-3be07c916d3c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a1dedaa8d09543c1a896888969b33f28', 'request_id': 'req-dcf83d7a-821c-439f-b23e-3be07c916d3c'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 04:05:50.370555 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-dcf83d7a-821c-439f-b23e-3be07c916d3c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a1dedaa8d09543c1a896888969b33f28', 'request_id': 'req-dcf83d7a-821c-439f-b23e-3be07c916d3c'} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 04:05:50.381924 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-dcf83d7a-821c-439f-b23e-3be07c916d3c admin admin] The token cache is being invalidate because project a1dedaa8d09543c1a896888969b33f28 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81761) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 04:05:50.382205 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-dcf83d7a-821c-439f-b23e-3be07c916d3c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 04:05:50.383397 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG keystone.notifications [None req-dcf83d7a-821c-439f-b23e-3be07c916d3c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81761) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 04:05:50.384980 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: [pid: 81761|app: 0|req: 1461/2931] 10.222.0.49 () {58 vars in 1233 bytes} [Fri Sep 16 04:05:50 2022] DELETE /identity/v3/projects/a1dedaa8d09543c1a896888969b33f28 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 04:05:59.858897 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=52,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 04:06:00.445213 dv-cinder-857993-1-iscsi devstack@keystone.service[81761]: DEBUG dbcounter [-] [81761] Writing DB stats keystone:SELECT=16,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81761) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 04:06:33.721954 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e1374d-ed87-47d9-8138-818465b03c34 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 04:06:33.723435 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d3e1374d-ed87-47d9-8138-818465b03c34 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 04:06:33.813310 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e1374d-ed87-47d9-8138-818465b03c34 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 04:06:33.874764 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a563d215-1d2d-40d9-bd72-9b58da1b2f39 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '176da4679fcb46148da9289fccdad5c2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0bb7350474f349d6aee197ff35536167', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 04:06:33.875441 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a563d215-1d2d-40d9-bd72-9b58da1b2f39 service placement] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 04:06:33.875721 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a563d215-1d2d-40d9-bd72-9b58da1b2f39 service placement] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 04:06:33.875998 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a563d215-1d2d-40d9-bd72-9b58da1b2f39 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 04:06:33.877794 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a563d215-1d2d-40d9-bd72-9b58da1b2f39 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 04:06:33.889900 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a563d215-1d2d-40d9-bd72-9b58da1b2f39 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 04:06:33.891505 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a563d215-1d2d-40d9-bd72-9b58da1b2f39 service placement] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 04:06:33.901565 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1471/2932] 10.222.0.49 () {60 vars in 1365 bytes} [Fri Sep 16 04:06:33 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 04:06:43.897108 dv-cinder-857993-1-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=67 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}